[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:59 dist-test-slave-530n krb5kdc[2877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:59 dist-test-slave-530n krb5kdc[2877](info): set up 2 sockets
Sep 29 03:00:59 dist-test-slave-530n krb5kdc[2877](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:01:00.613435 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:00 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:00.786101 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:42751
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:01.104821 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:01.105139 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:01.105255 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:01.112830 2891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:01.112949 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:01.113006 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:01.113088 2891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:01.113152 2891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:01.123494 2891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:01.124933 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:01.126325 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:01.138525 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:01.138649 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:01.138715 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:01.139532 2891 server_base.cc:1034] running on GCE node
I20260929 03:01:01.140455 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:01.142212 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:01.143436 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790650861143381 us; error 46 us; skew 500 ppm
I20260929 03:01:01.143800 2891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:01 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:01.151238 2891 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:01.152999 2891 webserver.cc:460] Webserver started at http://127.2.205.190:38081/ using document root <none> and password file <none>
I20260929 03:01:01.153757 2891 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:01.153868 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:01.154191 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:01.157428 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "5d5f95fdc7f6483a984a2dae4da56bcc"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "feeb932428c098b57a67de46ea06e66f"
server_key_iv: "0b0121d9a48f85996aff74dd73bd8074"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.158219 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "5d5f95fdc7f6483a984a2dae4da56bcc"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "feeb932428c098b57a67de46ea06e66f"
server_key_iv: "0b0121d9a48f85996aff74dd73bd8074"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.163758 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:01.167661 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:01.169299 2891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:01:01.169471 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "5d5f95fdc7f6483a984a2dae4da56bcc"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "feeb932428c098b57a67de46ea06e66f"
server_key_iv: "0b0121d9a48f85996aff74dd73bd8074"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.169672 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:01.195801 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:01.200136 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:01.200407 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:01.217834 2891 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:42751
I20260929 03:01:01.217844 2957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:42751 every 8 connection(s)
I20260929 03:01:01.219244 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:01.223071 2958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:01.223773 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 2891
I20260929 03:01:01.224094 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:01.226166 2870 external_mini_cluster.cc:1428] Setting key d4c1b90e02eab29f504df46cc02ccc45
I20260929 03:01:01.236791 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc: Bootstrap starting.
I20260929 03:01:01.241387 2958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:01.242713 2958 log.cc:824] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:01.245929 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc: No bootstrap required, opened a new log
I20260929 03:01:01.255743 2958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } }
I20260929 03:01:01.256145 2958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:01.256263 2958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5f95fdc7f6483a984a2dae4da56bcc, State: Initialized, Role: FOLLOWER
I20260929 03:01:01.257019 2958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } }
I20260929 03:01:01.257328 2958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:01.257476 2958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:01.257661 2958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:01.259806 2958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } }
I20260929 03:01:01.260298 2958 leader_election.cc:302] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d5f95fdc7f6483a984a2dae4da56bcc; no voters:
I20260929 03:01:01.260923 2958 leader_election.cc:288] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:01.261030 2963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 1 FOLLOWER]: Leader election won for term 1
Sep 29 03:01:01 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:01.263223 2958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:01.263289 2963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [term 1 LEADER]: Becoming Leader. State: Replica: 5d5f95fdc7f6483a984a2dae4da56bcc, State: Running, Role: LEADER
I20260929 03:01:01.264041 2963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } }
I20260929 03:01:01.267642 2964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } } }
I20260929 03:01:01.267650 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d5f95fdc7f6483a984a2dae4da56bcc. Latest consensus state: current_term: 1 leader_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5f95fdc7f6483a984a2dae4da56bcc" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42751 } } }
I20260929 03:01:01.268127 2964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:01.268159 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:01.274811 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:01.230191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.2.205.190:42751)
0929 03:01:01.235729 (+ 5538us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:01.235750 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:01.237355 (+ 1605us) server_negotiation.cc:407] Connection header received
0929 03:01:01.248236 (+ 10881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:01.248266 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:01.248866 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:01.249381 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:01.252300 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.254393 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.256133 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.257200 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.257404 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:01.263196 (+ 5792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:01.263212 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:01.263232 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:01.263301 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:01.267231 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:01.268265 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:01.268274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:01.268285 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:01.268542 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:01.269133 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:01.269137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:01.269140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:01.269929 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:01.270266 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:01.273101 (+ 2835us) server_negotiation.cc:299] Negotiation successful
0929 03:01:01.273433 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5251,"thread_start_us":4997,"threads_started":1}
I20260929 03:01:01.276448 2972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:01.281944 2972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:01.292153 2972 catalog_manager.cc:1347] Generated new cluster ID: 1dd5238432144821970cf2b4b29cdfa7
I20260929 03:01:01.292282 2972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:01.304689 2972 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:01.306133 2972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:01.312844 2972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d5f95fdc7f6483a984a2dae4da56bcc: Generated new TSK 0
I20260929 03:01:01.314033 2972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:01.360512 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:01.690366 2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:01.690680 2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:01.690832 2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:01.698803 2984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:01.698961 2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:01.699123 2984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:01.710011 2984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:01.711601 2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:01.713089 2984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:01.727563 2989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:01.727528 2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:01.727499 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:01.728246 2984 server_base.cc:1034] running on GCE node
I20260929 03:01:01.728916 2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:01.729948 2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:01.731184 2984 hybrid_clock.cc:648] HybridClock initialized: now 1790650861731118 us; error 61 us; skew 500 ppm
I20260929 03:01:01.731488 2984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:01 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:01.734723 2984 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:01.736290 2984 webserver.cc:460] Webserver started at http://127.2.205.129:37143/ using document root <none> and password file <none>
I20260929 03:01:01.737069 2984 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:01.737178 2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:01.737454 2984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:01.740468 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "161472b8a9904b25bcff3d1ac27dc1a7"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "bf8504f5df6cb10ca08901243bbdde38"
server_key_iv: "ec75fa29d812fa8f191ca9ae212555e1"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.741412 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "161472b8a9904b25bcff3d1ac27dc1a7"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "bf8504f5df6cb10ca08901243bbdde38"
server_key_iv: "ec75fa29d812fa8f191ca9ae212555e1"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.747061 2984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:01.750831 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:01.752219 2984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:01:01.752388 2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "161472b8a9904b25bcff3d1ac27dc1a7"
format_stamp: "Formatted at 2026-09-29 03:01:01 on dist-test-slave-530n"
server_key: "bf8504f5df6cb10ca08901243bbdde38"
server_key_iv: "ec75fa29d812fa8f191ca9ae212555e1"
server_key_version: "encryptionkey@0"
I20260929 03:01:01.752570 2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:01.769568 2984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:01.773780 2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:01.774044 2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:01.774952 2984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:01.776328 2984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:01.776400 2984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:01.776518 2984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:01.776571 2984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:01.805924 2984 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37107
I20260929 03:01:01.805970 3111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37107 every 8 connection(s)
I20260929 03:01:01.807261 2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:01.808027 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 2984
I20260929 03:01:01.808296 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:01.809051 2870 external_mini_cluster.cc:1428] Setting key 95af2edff5469b268aa32b0e1197f412
Sep 29 03:01:01 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650861, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:01.830041 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:01.810207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:33843 (local address 127.2.205.190:42751)
0929 03:01:01.810692 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:01.810698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:01.811903 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 03:01:01.814269 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:01.814275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:01.814362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:01.814485 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:01.816349 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.817474 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.819171 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.819767 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.819903 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:01.823694 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:01.823702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:01.823708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:01.823765 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:01.826525 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:01.827490 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:01.827496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:01.827498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:01.827639 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:01.828139 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:01.828142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:01.828144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:01.828420 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:01.828550 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:01.829574 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 03:01:01.829771 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":245,"threads_started":1}
I20260929 03:01:01.831565 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:01.810905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42751 (local address 127.2.205.129:33843)
0929 03:01:01.811731 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:01.811776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:01.813949 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:01.814703 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:01.814801 (+ 98us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:01.815514 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:01.816153 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:01.816173 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.817672 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.817676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:01.819017 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:01.819030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:01.820008 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:01.820013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:01.820309 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:01.821077 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:01.821106 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:01.823402 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:01.826712 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:01.826720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:01.826738 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:01.827330 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:01.827765 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:01.827770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:01.827774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:01.828035 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:01.828542 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:01.828550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:01.829104 (+ 554us) client_negotiation.cc:769] Sending connection context
0929 03:01:01.829444 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 03:01:01.829999 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260929 03:01:01.833302 3112 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42751
I20260929 03:01:01.833603 3112 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:01.834378 3112 heartbeater.cc:511] Master 127.2.205.190:42751 requested a full tablet report, sending...
I20260929 03:01:01.836879 2923 ts_manager.cc:194] Registered new tserver with Master: 161472b8a9904b25bcff3d1ac27dc1a7 (127.2.205.129:37107)
I20260929 03:01:01.839013 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:33843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:01.848331 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:02.171990 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:02.172310 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:02.172441 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:02.179915 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:02.180063 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:02.180214 3118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:02.190569 3118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:02.191989 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:02.193437 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:02.206913 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:02.207063 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:02.206948 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:02.207389 3118 server_base.cc:1034] running on GCE node
I20260929 03:01:02.208418 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:02.209532 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:02.210776 3118 hybrid_clock.cc:648] HybridClock initialized: now 1790650862210700 us; error 60 us; skew 500 ppm
I20260929 03:01:02.211133 3118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:02.214488 3118 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:02.216159 3118 webserver.cc:460] Webserver started at http://127.2.205.130:39781/ using document root <none> and password file <none>
I20260929 03:01:02.216960 3118 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:02.217092 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:02.217455 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:02.220571 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "b46b426883544c19894b9f57c5f5a85c"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "d78f37ebacf75dda9138093a34b74bf4"
server_key_iv: "246f4ed826cdc6db392015ecdf11fa06"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.221483 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "b46b426883544c19894b9f57c5f5a85c"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "d78f37ebacf75dda9138093a34b74bf4"
server_key_iv: "246f4ed826cdc6db392015ecdf11fa06"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.227447 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 03:01:02.231911 3133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.233734 3118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:01:02.233927 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "b46b426883544c19894b9f57c5f5a85c"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "d78f37ebacf75dda9138093a34b74bf4"
server_key_iv: "246f4ed826cdc6db392015ecdf11fa06"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.234139 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:02.251380 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:02.255445 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:02.255713 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:02.256682 3118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:02.258021 3118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:02.258092 3118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.258199 3118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:02.258265 3118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.289654 3118 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:45473
I20260929 03:01:02.289685 3245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:45473 every 8 connection(s)
I20260929 03:01:02.291060 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:02.293689 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3118
I20260929 03:01:02.293885 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:02.294317 2870 external_mini_cluster.cc:1428] Setting key fda51dc186dd77f0bb1223101e9d61de
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:02.312697 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.293797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:36147 (local address 127.2.205.190:42751)
0929 03:01:02.294025 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:02.294031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:02.295328 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 03:01:02.297306 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:02.297312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:02.297411 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:02.297499 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:02.299184 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.299981 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.301700 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.302157 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.302277 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.306386 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:02.306392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:02.306395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:02.306441 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:02.309316 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.310242 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.310247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.310250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.310390 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.310891 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.310895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.310897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.311185 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:02.311326 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:02.312091 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 03:01:02.312353 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 03:01:02.313833 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.294304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42751 (local address 127.2.205.130:36147)
0929 03:01:02.295182 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:02.295231 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:02.297023 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:02.297783 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:02.297792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:02.298349 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:02.298987 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:02.299007 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.300159 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.300163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:02.301534 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:02.301548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.302407 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.302416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:02.302725 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.303537 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:02.303563 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:02.306096 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:02.309493 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:02.309505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:02.309523 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:02.310082 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:02.310513 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:02.310518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:02.310522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:02.310765 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:02.311326 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:02.311336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:02.311715 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 03:01:02.311969 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 03:01:02.312538 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":340,"threads_started":1}
I20260929 03:01:02.315551 3246 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42751
I20260929 03:01:02.315865 3246 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:02.316634 3246 heartbeater.cc:511] Master 127.2.205.190:42751 requested a full tablet report, sending...
I20260929 03:01:02.318285 2923 ts_manager.cc:194] Registered new tserver with Master: b46b426883544c19894b9f57c5f5a85c (127.2.205.130:45473)
I20260929 03:01:02.319162 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:36147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:02.332023 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:02.660175 3251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:02.660462 3251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:02.660562 3251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:02.667830 3251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:02.667934 3251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:02.668048 3251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:02.678062 3251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:02.679348 3251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:02.680670 3251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:02.694377 3257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:02.694433 3259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:02.694406 3256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:02.695137 3251 server_base.cc:1034] running on GCE node
I20260929 03:01:02.695847 3251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:02.696885 3251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:02.698112 3251 hybrid_clock.cc:648] HybridClock initialized: now 1790650862698056 us; error 43 us; skew 500 ppm
I20260929 03:01:02.698478 3251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:02.701769 3251 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:02.703325 3251 webserver.cc:460] Webserver started at http://127.2.205.131:45933/ using document root <none> and password file <none>
I20260929 03:01:02.704031 3251 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:02.704133 3251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:02.704406 3251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:02.707432 3251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "a0168c342b1f45728b10c047ee3af885"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "f8dee2adadafed07eefebad750068c27"
server_key_iv: "06f5ab3baefe18879b07a159b4406580"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.708165 3251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "a0168c342b1f45728b10c047ee3af885"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "f8dee2adadafed07eefebad750068c27"
server_key_iv: "06f5ab3baefe18879b07a159b4406580"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.713657 3251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 03:01:02.717398 3266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.718936 3251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:01:02.719115 3251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "a0168c342b1f45728b10c047ee3af885"
format_stamp: "Formatted at 2026-09-29 03:01:02 on dist-test-slave-530n"
server_key: "f8dee2adadafed07eefebad750068c27"
server_key_iv: "06f5ab3baefe18879b07a159b4406580"
server_key_version: "encryptionkey@0"
I20260929 03:01:02.719353 3251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:02.738131 3251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:02.742445 3251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:02.742714 3251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:02.743686 3251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:02.745065 3251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:02.745163 3251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.745288 3251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:02.745348 3251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:02.775792 3251 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:44241
I20260929 03:01:02.775847 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:44241 every 8 connection(s)
I20260929 03:01:02.777056 3251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:02.778411 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3251
I20260929 03:01:02.778579 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:02.779014 2870 external_mini_cluster.cc:1428] Setting key d2f4c8878785c72dc4d490fd7a2ca60d
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:02.795789 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.779574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:44737 (local address 127.2.205.190:42751)
0929 03:01:02.779733 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:02.779737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:02.781179 (+ 1442us) server_negotiation.cc:407] Connection header received
0929 03:01:02.782932 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:02.782937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:02.783039 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:02.783124 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:02.784593 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.785383 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.786853 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.787249 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.787354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.790501 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:02.790506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:02.790509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:02.790542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:02.792926 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.793587 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.793593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.793595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.793717 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.794103 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.794108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.794111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.794402 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:02.794530 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:02.795208 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 03:01:02.795432 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:01:02.796836 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.780278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42751 (local address 127.2.205.131:44737)
0929 03:01:02.781031 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:02.781074 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:02.782671 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:02.783366 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:02.783375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:02.783861 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:02.784406 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:02.784423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.785533 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.785537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:02.786719 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:02.786731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.787455 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.787462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:02.787680 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.788244 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:02.788269 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:02.790300 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:02.793051 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:02.793066 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:02.793080 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:02.793472 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:02.793821 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:02.793825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:02.793829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:02.793996 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:02.794520 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:02.794527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:02.794848 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 03:01:02.795090 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 03:01:02.795603 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":285,"threads_started":1}
I20260929 03:01:02.798390 3379 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42751
I20260929 03:01:02.798722 3379 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:02.799557 3379 heartbeater.cc:511] Master 127.2.205.190:42751 requested a full tablet report, sending...
I20260929 03:01:02.801280 2923 ts_manager.cc:194] Registered new tserver with Master: a0168c342b1f45728b10c047ee3af885 (127.2.205.131:44241)
I20260929 03:01:02.802186 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:44737
I20260929 03:01:02.807359 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 03:01:02.842370 3112 heartbeater.cc:503] Master 127.2.205.190:42751 was elected leader, sending a full tablet report...
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:02.851114 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.835007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.2.205.129:37107)
0929 03:01:02.835708 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:02.835716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:02.835745 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:01:02.835891 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:02.835898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:02.836117 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:02.836282 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:02.837784 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.838902 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.840230 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.840898 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.841017 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.845387 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:02.845400 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:02.845416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:02.845455 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:02.848178 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.848920 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.848928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.848930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.849061 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.849518 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.849522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.849524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.849887 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:02.850167 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:02.850522 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 03:01:02.850779 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":381,"threads_started":1}
W20260929 03:01:02.852358 3027 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44272
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:01:02 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:02.894086 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.880339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.2.205.129:37107)
0929 03:01:02.880577 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:02.880585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:02.880771 (+ 186us) server_negotiation.cc:407] Connection header received
0929 03:01:02.880973 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:02.880977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:02.881046 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:02.881124 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:02.882637 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.883411 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.884703 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.885257 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.885357 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.889438 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:02.889444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:02.889447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:02.889488 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:02.891857 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.892466 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.892470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.892472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.892596 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.892928 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.892931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.892933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.893184 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:02.893360 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:02.893590 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 03:01:02.893776 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 03:01:02.895566 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 2984
I20260929 03:01:02.907689 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:02.896540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.2.205.129:37107)
0929 03:01:02.896807 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:02.896812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:02.897004 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:01:02.897150 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:02.897154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:02.897209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:02.897292 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:02.898427 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.899114 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.900358 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:02.900822 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:02.900929 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:02.903067 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:02.903072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:02.903075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:02.903105 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:02.905351 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.905938 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.905943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.905947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.906076 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:02.906412 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:02.906415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:02.906417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:02.906662 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:02.906864 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:02.907161 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 03:01:02.907353 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:02.908398 3107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:02.990540 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 2984
I20260929 03:01:03.003494 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3118
Sep 29 03:01:03 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:03.019032 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:03.004397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.2.205.130:45473)
0929 03:01:03.005000 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:03.005008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:03.005039 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:01:03.005101 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:03.005108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:03.005334 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:03.005524 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:03.006718 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:03.007776 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:03.009075 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:03.009695 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:03.009813 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:03.013651 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:03.013664 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:03.013679 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:03.013732 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:03.016435 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:03.017170 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:03.017178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:03.017180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:03.017317 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:03.017692 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:03.017696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:03.017698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:03.018055 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:03.018308 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:03.018523 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:01:03.018735 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":283,"threads_started":1}
I20260929 03:01:03.020309 3241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:03.103588 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3118
I20260929 03:01:03.116571 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3251
Sep 29 03:01:03 dist-test-slave-530n krb5kdc[2877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:03.134627 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:03.117712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.2.205.131:44241)
0929 03:01:03.118468 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:03.118475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:03.118506 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:01:03.118574 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:03.118580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:03.118806 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:03.118994 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:03.120367 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:03.121571 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:03.123018 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:03.123646 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:03.123808 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:03.128263 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:03.128280 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:03.128296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:03.128342 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:03.131400 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:03.132275 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:03.132284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:03.132287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:03.132446 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:03.132908 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:03.132913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:03.132915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:03.133320 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:03.133632 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:03.133981 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 03:01:03.134278 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260929 03:01:03.136029 3374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:03.219404 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3251
I20260929 03:01:03.232319 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 2891
I20260929 03:01:03.233708 2953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:03.339033 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 2891
2026-09-29T03:01:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3733 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:03 dist-test-slave-530n krb5kdc[3400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:03 dist-test-slave-530n krb5kdc[3400](info): set up 2 sockets
Sep 29 03:01:03 dist-test-slave-530n krb5kdc[3400](info): commencing operation
krb5kdc: starting...
W20260929 03:01:04.332892 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:04 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:04.500424 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35339
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:04.814463 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:04.814793 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:04.814930 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:04.822418 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:04.822546 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:04.822592 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:04.822674 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:04.822731 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:04.832965 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:04.834344 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:04.835655 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:04.847244 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:04.847438 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:04.847239 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:04.847977 3413 server_base.cc:1034] running on GCE node
I20260929 03:01:04.848698 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:04.850131 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:04.851323 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790650864851265 us; error 48 us; skew 500 ppm
I20260929 03:01:04.851626 3413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:04 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650864, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:04.854602 3413 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:04.855947 3413 webserver.cc:460] Webserver started at http://127.2.205.190:43063/ using document root <none> and password file <none>
I20260929 03:01:04.856586 3413 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:04.856683 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:04.856985 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:04.859817 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "979bd11ed9c04d88bc78452dabd86e57"
format_stamp: "Formatted at 2026-09-29 03:01:04 on dist-test-slave-530n"
server_key: "8e60e95655a76e1d5100ea5120d234bc"
server_key_iv: "92430f9b78fa4534df2e515a2ad5a1ba"
server_key_version: "encryptionkey@0"
I20260929 03:01:04.860481 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "979bd11ed9c04d88bc78452dabd86e57"
format_stamp: "Formatted at 2026-09-29 03:01:04 on dist-test-slave-530n"
server_key: "8e60e95655a76e1d5100ea5120d234bc"
server_key_iv: "92430f9b78fa4534df2e515a2ad5a1ba"
server_key_version: "encryptionkey@0"
I20260929 03:01:04.865593 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 03:01:04.869886 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:04.871526 3413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:01:04.871699 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "979bd11ed9c04d88bc78452dabd86e57"
format_stamp: "Formatted at 2026-09-29 03:01:04 on dist-test-slave-530n"
server_key: "8e60e95655a76e1d5100ea5120d234bc"
server_key_iv: "92430f9b78fa4534df2e515a2ad5a1ba"
server_key_version: "encryptionkey@0"
I20260929 03:01:04.871984 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:04.889111 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:04.895287 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:04.895615 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:04.914212 3413 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35339
I20260929 03:01:04.914265 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35339 every 8 connection(s)
I20260929 03:01:04.915650 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:04.916903 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3413
I20260929 03:01:04.917106 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:04.917721 2870 external_mini_cluster.cc:1428] Setting key a44ac37c7f8d44377b2ac07b0af81e96
I20260929 03:01:04.920241 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:04.933598 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57: Bootstrap starting.
Sep 29 03:01:04 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:04.937595 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:04.939000 3481 log.cc:824] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:04.942844 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57: No bootstrap required, opened a new log
I20260929 03:01:04.944283 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:04.919786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.2.205.190:35339)
0929 03:01:04.920584 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:04.920599 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:04.920650 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:01:04.922945 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:04.922982 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:04.923562 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:04.923999 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:04.926078 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:04.928114 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:04.929680 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:04.930844 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:04.931042 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:04.935079 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:04.935090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:04.935116 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:04.935188 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:04.939139 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:04.940188 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:04.940197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:04.940204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:04.940414 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:04.940861 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:04.940867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:04.940870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:04.941615 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:04.941912 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:04.942395 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 03:01:04.942871 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":328,"threads_started":1}
I20260929 03:01:04.951334 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } }
I20260929 03:01:04.951669 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:04.951740 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979bd11ed9c04d88bc78452dabd86e57, State: Initialized, Role: FOLLOWER
I20260929 03:01:04.952253 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } }
I20260929 03:01:04.952450 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:04.952540 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:04.952662 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:04.954759 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } }
I20260929 03:01:04.955276 3481 leader_election.cc:302] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979bd11ed9c04d88bc78452dabd86e57; no voters:
I20260929 03:01:04.955874 3481 leader_election.cc:288] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:04.955968 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:04.956597 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [term 1 LEADER]: Becoming Leader. State: Replica: 979bd11ed9c04d88bc78452dabd86e57, State: Running, Role: LEADER
I20260929 03:01:04.957115 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } }
I20260929 03:01:04.957828 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:04.959478 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979bd11ed9c04d88bc78452dabd86e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } } }
I20260929 03:01:04.959801 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:04.960542 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:04.964800 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979bd11ed9c04d88bc78452dabd86e57. Latest consensus state: current_term: 1 leader_uuid: "979bd11ed9c04d88bc78452dabd86e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979bd11ed9c04d88bc78452dabd86e57" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35339 } } }
I20260929 03:01:04.965193 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:04.971496 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:04.979463 3493 catalog_manager.cc:1347] Generated new cluster ID: f6904cd133a544ee8f2d386a79c15d15
I20260929 03:01:04.979564 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:04.993345 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:04.994843 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:05.002499 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979bd11ed9c04d88bc78452dabd86e57: Generated new TSK 0
I20260929 03:01:05.003486 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:05.055830 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:05.410317 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:05.410614 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:05.410753 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:05.418063 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:05.418192 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:05.418354 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:05.428702 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:05.430153 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:05.431529 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:05.444706 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:05.444657 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:05.444989 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:05.445353 3508 server_base.cc:1034] running on GCE node
I20260929 03:01:05.445988 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:05.446938 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:05.448151 3508 hybrid_clock.cc:648] HybridClock initialized: now 1790650865448088 us; error 56 us; skew 500 ppm
I20260929 03:01:05.448455 3508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:05 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650865, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:05.451459 3508 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:05.452913 3508 webserver.cc:460] Webserver started at http://127.2.205.129:39823/ using document root <none> and password file <none>
I20260929 03:01:05.453594 3508 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:05.453715 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:05.454005 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:05.457137 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "229fb7c6fa644267966cbd2716b51df4"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "f9d16c80b85e76def6635ae7f87c995d"
server_key_iv: "9abfbaa54be7b6e6d566256576c60ef1"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.457926 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "229fb7c6fa644267966cbd2716b51df4"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "f9d16c80b85e76def6635ae7f87c995d"
server_key_iv: "9abfbaa54be7b6e6d566256576c60ef1"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.463236 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:05.466650 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:05.468062 3508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:01:05.468201 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "229fb7c6fa644267966cbd2716b51df4"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "f9d16c80b85e76def6635ae7f87c995d"
server_key_iv: "9abfbaa54be7b6e6d566256576c60ef1"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.468387 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:05.483857 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:05.488027 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:05.488301 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:05.489400 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:05.490990 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:05.491068 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:05.491173 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:05.491235 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:05.520988 3508 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:42491
I20260929 03:01:05.521034 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:42491 every 8 connection(s)
I20260929 03:01:05.522293 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:05.524907 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3508
I20260929 03:01:05.525125 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:05.525899 2870 external_mini_cluster.cc:1428] Setting key d3fb46aa92745cf4dc4970cdd256b377
Sep 29 03:01:05 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650865, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:05.545113 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:05.524849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:46473 (local address 127.2.205.190:35339)
0929 03:01:05.525402 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:05.525406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:05.526570 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 03:01:05.528566 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:05.528571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:05.528665 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:05.528782 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:05.530427 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:05.531128 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:05.533274 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:05.533869 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:05.533996 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:05.537955 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:05.537964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:05.537969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:05.538018 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:05.541515 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:05.542582 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:05.542588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:05.542591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:05.542737 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:05.543253 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:05.543257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:05.543259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:05.543545 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:05.543659 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:05.544484 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 03:01:05.544781 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":325,"threads_started":1}
I20260929 03:01:05.546257 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:05.525524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35339 (local address 127.2.205.129:46473)
0929 03:01:05.526390 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:05.526435 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:05.528277 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:05.529052 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:05.529062 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:05.529615 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:05.530240 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:05.530258 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:05.531304 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:05.531308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:05.533114 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:05.533137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:05.534104 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:05.534109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:05.534406 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:05.535157 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:05.535182 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:05.537578 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:05.541720 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:05.541730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:05.541747 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:05.542395 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:05.542843 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:05.542847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:05.542849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:05.543121 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:05.543644 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:05.543651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:05.544069 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 03:01:05.544307 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 03:01:05.544841 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":358,"threads_started":1}
I20260929 03:01:05.547964 3637 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35339
I20260929 03:01:05.548277 3637 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:05.549081 3637 heartbeater.cc:511] Master 127.2.205.190:35339 requested a full tablet report, sending...
I20260929 03:01:05.551256 3446 ts_manager.cc:194] Registered new tserver with Master: 229fb7c6fa644267966cbd2716b51df4 (127.2.205.129:42491)
I20260929 03:01:05.552716 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:46473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:05.567406 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:05.885691 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:05.885987 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:05.886088 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:05.893501 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:05.893620 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:05.893746 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:05.903594 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:05.904953 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:05.906250 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:05.920533 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:05.920780 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:05.920845 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:05.921725 3643 server_base.cc:1034] running on GCE node
I20260929 03:01:05.922463 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:05.923660 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:05.924847 3643 hybrid_clock.cc:648] HybridClock initialized: now 1790650865924777 us; error 59 us; skew 500 ppm
I20260929 03:01:05.925215 3643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:05 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650865, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:05.929155 3643 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:05.930964 3643 webserver.cc:460] Webserver started at http://127.2.205.130:35453/ using document root <none> and password file <none>
I20260929 03:01:05.931916 3643 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:05.932050 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:05.932441 3643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:05.936093 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "cdbf7faa3d914318a6f8a7817f37c31c"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "bf012a1f2570c38e78912af527ec1765"
server_key_iv: "381464fd9fdf4f2b3d26f81865b4bb43"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.937000 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "cdbf7faa3d914318a6f8a7817f37c31c"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "bf012a1f2570c38e78912af527ec1765"
server_key_iv: "381464fd9fdf4f2b3d26f81865b4bb43"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.942740 3643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:05.946401 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:05.947808 3643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:05.947956 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "cdbf7faa3d914318a6f8a7817f37c31c"
format_stamp: "Formatted at 2026-09-29 03:01:05 on dist-test-slave-530n"
server_key: "bf012a1f2570c38e78912af527ec1765"
server_key_iv: "381464fd9fdf4f2b3d26f81865b4bb43"
server_key_version: "encryptionkey@0"
I20260929 03:01:05.948150 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:05.971822 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:05.975879 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:05.976148 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:05.977175 3643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:05.978514 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:05.978582 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:05.978688 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:05.978734 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:06.009433 3643 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:39637
I20260929 03:01:06.009485 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:39637 every 8 connection(s)
I20260929 03:01:06.010730 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:06.013710 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3643
I20260929 03:01:06.013942 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:06.014516 2870 external_mini_cluster.cc:1428] Setting key 952b00350f5ae9a452bb00df0dc63d4f
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650865, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:06.032069 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.013293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:58027 (local address 127.2.205.190:35339)
0929 03:01:06.013475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.013479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.014984 (+ 1505us) server_negotiation.cc:407] Connection header received
0929 03:01:06.017326 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.017332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.017414 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.017501 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.019091 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.019813 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.021446 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.021883 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.021997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.025842 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.025851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.025857 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.025906 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.028551 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.029634 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.029641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.029645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.029793 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.030229 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.030232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.030234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.030562 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.030680 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.031536 (+ 856us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.031780 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:06.033308 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.013964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35339 (local address 127.2.205.130:58027)
0929 03:01:06.014804 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.014852 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.017042 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.017770 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.017780 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.018308 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:06.018915 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.018933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.019971 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.019975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.021295 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.021308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.022115 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.022143 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.022382 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.023057 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:06.023082 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:06.025560 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:06.028775 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:06.028787 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:06.028808 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:06.029466 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:06.029907 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:06.029910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:06.029913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:06.030114 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:06.030699 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:06.030713 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:06.031070 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.031345 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.031832 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":313,"threads_started":1}
I20260929 03:01:06.034957 3771 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35339
I20260929 03:01:06.035275 3771 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:06.036023 3771 heartbeater.cc:511] Master 127.2.205.190:35339 requested a full tablet report, sending...
I20260929 03:01:06.037639 3446 ts_manager.cc:194] Registered new tserver with Master: cdbf7faa3d914318a6f8a7817f37c31c (127.2.205.130:39637)
I20260929 03:01:06.038465 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:58027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:06.052048 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:06.370180 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:06.370496 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:06.370626 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:06.378580 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:06.378731 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:06.378898 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:06.389007 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:06.390377 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:06.391703 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:06.404840 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:06.404778 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:06.404842 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:06.405426 3776 server_base.cc:1034] running on GCE node
I20260929 03:01:06.406044 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:06.406965 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:06.408174 3776 hybrid_clock.cc:648] HybridClock initialized: now 1790650866408121 us; error 38 us; skew 500 ppm
I20260929 03:01:06.408520 3776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:06.414630 3776 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:06.416266 3776 webserver.cc:460] Webserver started at http://127.2.205.131:46337/ using document root <none> and password file <none>
I20260929 03:01:06.417066 3776 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:06.417176 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:06.417461 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:06.420516 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "957b103a2ab84305b7124c422a1ba394"
format_stamp: "Formatted at 2026-09-29 03:01:06 on dist-test-slave-530n"
server_key: "3d245e50664dea9fc312ba96e84d76c5"
server_key_iv: "4ef9924ad3a09b2625e9e7672a6d378d"
server_key_version: "encryptionkey@0"
I20260929 03:01:06.421414 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "957b103a2ab84305b7124c422a1ba394"
format_stamp: "Formatted at 2026-09-29 03:01:06 on dist-test-slave-530n"
server_key: "3d245e50664dea9fc312ba96e84d76c5"
server_key_iv: "4ef9924ad3a09b2625e9e7672a6d378d"
server_key_version: "encryptionkey@0"
I20260929 03:01:06.427376 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 03:01:06.431157 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:06.432670 3776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:01:06.432879 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "957b103a2ab84305b7124c422a1ba394"
format_stamp: "Formatted at 2026-09-29 03:01:06 on dist-test-slave-530n"
server_key: "3d245e50664dea9fc312ba96e84d76c5"
server_key_iv: "4ef9924ad3a09b2625e9e7672a6d378d"
server_key_version: "encryptionkey@0"
I20260929 03:01:06.433087 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:06.448668 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:06.452714 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:06.453014 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:06.453975 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:06.455284 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:06.455354 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:06.455461 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:06.455510 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:06.484980 3776 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:36881
I20260929 03:01:06.485014 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:36881 every 8 connection(s)
I20260929 03:01:06.486354 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:06.487511 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3776
I20260929 03:01:06.487700 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:06.488237 2870 external_mini_cluster.cc:1428] Setting key 170e747a4c67c0b5e93890bcc2675cef
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:06.508091 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.489362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:33875 (local address 127.2.205.190:35339)
0929 03:01:06.489576 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.489582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.490980 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 03:01:06.493059 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.493064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.493181 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.493267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.494928 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.495677 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.497364 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.497781 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.497901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.501613 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.501620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.501623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.501661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.505086 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.505963 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.505966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.505969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.506106 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.506506 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.506509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.506511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.506782 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.506901 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.507619 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.507804 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:01:06.509191 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.489996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35339 (local address 127.2.205.131:33875)
0929 03:01:06.490830 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.490868 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.492743 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.493550 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.493560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.494110 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:06.494735 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.494754 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.495854 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.495858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.497199 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.497214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.498033 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.498041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.498296 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.498968 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:06.498993 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:06.501338 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:06.505279 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:06.505288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:06.505304 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:06.505847 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:06.506193 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:06.506196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:06.506198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:06.506414 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:06.506906 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:06.506913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:06.507291 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.507534 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.507981 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":351,"threads_started":1}
I20260929 03:01:06.510664 3904 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35339
I20260929 03:01:06.510977 3904 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:06.511773 3904 heartbeater.cc:511] Master 127.2.205.190:35339 requested a full tablet report, sending...
I20260929 03:01:06.513464 3446 ts_manager.cc:194] Registered new tserver with Master: 957b103a2ab84305b7124c422a1ba394 (127.2.205.131:36881)
I20260929 03:01:06.514281 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:33875
I20260929 03:01:06.517279 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:01:06.536003 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.523683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.2.205.190:35339)
0929 03:01:06.523917 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.523923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.524319 (+ 396us) server_negotiation.cc:407] Connection header received
0929 03:01:06.524479 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.524485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.524568 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.524686 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.526123 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.526876 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.528079 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.528462 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.528556 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.530855 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.530861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.530864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.530901 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.533554 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.534337 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.534342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.534344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.534485 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.534801 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.534804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.534806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.535124 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.535308 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.535569 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.535758 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:01:06.542820 3446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:06.549817 3446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:06.556602 3637 heartbeater.cc:503] Master 127.2.205.190:35339 was elected leader, sending a full tablet report...
I20260929 03:01:06.577085 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.562483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.2.205.129:42491)
0929 03:01:06.563064 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.563074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.563701 (+ 627us) server_negotiation.cc:407] Connection header received
0929 03:01:06.563978 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.563985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.564247 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.564454 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:06.565412 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.568255 (+ 2843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.572459 (+ 4204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.575581 (+ 3122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.575765 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.576072 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.576462 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.576685 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":361,"threads_started":1}
I20260929 03:01:06.577116 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.562513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:42491 (local address 127.0.0.1:38482)
0929 03:01:06.563565 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.563599 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.563780 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.564716 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.564724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.564797 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:06.565208 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.565224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.568435 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.568441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.572240 (+ 3799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.572260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.575830 (+ 3570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.575838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.576105 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.576167 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.576338 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.576620 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":672,"threads_started":1}
I20260929 03:01:06.581367 3572 tablet_service.cc:1467] Processing CreateTablet for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7 (DEFAULT_TABLE table=test-table [id=03e7061d71ec4b8f9d56676d86a9fd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:06.582906 3572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:06.589802 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.564285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.2.205.131:36881)
0929 03:01:06.582976 (+ 18691us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.582984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.583016 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:01:06.583114 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.583121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.583330 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.583494 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:06.584077 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.585281 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.587401 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.588801 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.588930 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.589239 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.589413 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.589541 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18539,"thread_start_us":19423,"threads_started":1}
I20260929 03:01:06.589943 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.563940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:36881 (local address 127.0.0.1:52172)
0929 03:01:06.564419 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.564435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.564543 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.583522 (+ 18979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.583528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.583560 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:06.583906 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.583920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.585440 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.585445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.587231 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.587249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.589054 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.589061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.589232 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.589269 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.589373 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.589527 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":68,"thread_start_us":226,"threads_started":1}
I20260929 03:01:06.590966 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.564461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:39637 (local address 127.0.0.1:40872)
0929 03:01:06.574733 (+ 10272us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.574756 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.574897 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.582484 (+ 7587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.582490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.582524 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:06.582871 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.582887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.584879 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.584884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.587104 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.587123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.590166 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.590176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.590353 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.590385 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.590477 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.590605 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10122,"mutex_wait_us":57,"thread_start_us":9772,"threads_started":1}
I20260929 03:01:06.591020 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.564960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.2.205.130:39637)
0929 03:01:06.581834 (+ 16874us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.581844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.581875 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:01:06.581974 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.581982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.582247 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.582426 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:06.583116 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.584650 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.587277 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.588798 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.589050 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.589222 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.590548 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.590722 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16707,"thread_start_us":16592,"threads_started":1}
I20260929 03:01:06.594148 3839 tablet_service.cc:1467] Processing CreateTablet for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7 (DEFAULT_TABLE table=test-table [id=03e7061d71ec4b8f9d56676d86a9fd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:06.594668 3706 tablet_service.cc:1467] Processing CreateTablet for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7 (DEFAULT_TABLE table=test-table [id=03e7061d71ec4b8f9d56676d86a9fd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:06.595384 3839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:06.596028 3706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:06.597842 3922 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Bootstrap starting.
I20260929 03:01:06.601837 3922 tablet_bootstrap.cc:654] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:06.603333 3922 log.cc:824] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:06.606709 3922 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: No bootstrap required, opened a new log
I20260929 03:01:06.607007 3922 ts_tablet_manager.cc:1397] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 03:01:06.608207 3924 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Bootstrap starting.
I20260929 03:01:06.610308 3925 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Bootstrap starting.
I20260929 03:01:06.611876 3924 tablet_bootstrap.cc:654] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:06.613823 3925 tablet_bootstrap.cc:654] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:06.613811 3924 log.cc:824] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:06.614970 3925 log.cc:824] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:06.616775 3922 raft_consensus.cc:348] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.617095 3924 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: No bootstrap required, opened a new log
I20260929 03:01:06.617161 3922 raft_consensus.cc:374] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:06.617249 3922 raft_consensus.cc:729] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229fb7c6fa644267966cbd2716b51df4, State: Initialized, Role: FOLLOWER
I20260929 03:01:06.617403 3924 ts_tablet_manager.cc:1397] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 03:01:06.617882 3922 consensus_queue.cc:260] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.618044 3925 tablet_bootstrap.cc:492] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: No bootstrap required, opened a new log
I20260929 03:01:06.618290 3925 ts_tablet_manager.cc:1397] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 03:01:06.620819 3922 ts_tablet_manager.cc:1428] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260929 03:01:06.626677 3924 raft_consensus.cc:348] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.627084 3924 raft_consensus.cc:374] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:06.627182 3924 raft_consensus.cc:729] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdbf7faa3d914318a6f8a7817f37c31c, State: Initialized, Role: FOLLOWER
I20260929 03:01:06.627429 3925 raft_consensus.cc:348] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.627766 3925 raft_consensus.cc:374] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:06.627869 3925 raft_consensus.cc:729] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957b103a2ab84305b7124c422a1ba394, State: Initialized, Role: FOLLOWER
I20260929 03:01:06.627786 3924 consensus_queue.cc:260] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.628422 3925 consensus_queue.cc:260] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.628973 3771 heartbeater.cc:503] Master 127.2.205.190:35339 was elected leader, sending a full tablet report...
I20260929 03:01:06.629504 3904 heartbeater.cc:503] Master 127.2.205.190:35339 was elected leader, sending a full tablet report...
I20260929 03:01:06.629637 3924 ts_tablet_manager.cc:1428] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 03:01:06.630123 3925 ts_tablet_manager.cc:1428] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 03:01:06.678222 3929 raft_consensus.cc:482] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:06.678480 3929 raft_consensus.cc:504] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.680029 3929 leader_election.cc:288] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 229fb7c6fa644267966cbd2716b51df4 (127.2.205.129:42491), 957b103a2ab84305b7124c422a1ba394 (127.2.205.131:36881)
I20260929 03:01:06.687310 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.680547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:59483 (local address 127.2.205.129:42491)
0929 03:01:06.680790 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.680796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.681189 (+ 393us) server_negotiation.cc:407] Connection header received
0929 03:01:06.681454 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.681460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.681538 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.681603 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:06.682385 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.683224 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.685256 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.686246 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.686408 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.686491 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.686896 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.687040 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:06.687302 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.681473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:43121 (local address 127.2.205.131:36881)
0929 03:01:06.681657 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.681662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.681817 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:01:06.682014 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.682019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.682092 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.682205 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:06.682750 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.683542 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.685357 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.686357 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.686512 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.686582 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.686954 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.687077 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:01:06.687471 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.681187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:36881 (local address 127.2.205.130:43121)
0929 03:01:06.681719 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.681734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.681888 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.682227 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.682230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.682249 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:06.682610 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.682622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.683685 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.683688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.685171 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.685186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.686639 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.686644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.686808 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.686848 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.686919 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.687020 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"spinlock_wait_cycles":20224,"thread_start_us":265,"threads_started":1}
I20260929 03:01:06.687840 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.680418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:42491 (local address 127.2.205.130:59483)
0929 03:01:06.681083 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:06.681099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:06.681249 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:06.681789 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:06.681794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:06.681821 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:06.682223 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.682236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.683376 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.683382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.685082 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:06.685099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.686515 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.686521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:06.686714 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.686753 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:01:06.686854 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:01:06.687008 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":390,"threads_started":1}
I20260929 03:01:06.688495 3859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff53fc68e4a14ecebe0e0ff08b5b85a7" candidate_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957b103a2ab84305b7124c422a1ba394" is_pre_election: true
I20260929 03:01:06.688741 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff53fc68e4a14ecebe0e0ff08b5b85a7" candidate_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229fb7c6fa644267966cbd2716b51df4" is_pre_election: true
I20260929 03:01:06.689031 3859 raft_consensus.cc:2454] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbf7faa3d914318a6f8a7817f37c31c in term 0.
I20260929 03:01:06.689303 3592 raft_consensus.cc:2454] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdbf7faa3d914318a6f8a7817f37c31c in term 0.
I20260929 03:01:06.689715 3662 leader_election.cc:302] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957b103a2ab84305b7124c422a1ba394, cdbf7faa3d914318a6f8a7817f37c31c; no voters:
I20260929 03:01:06.690092 3929 raft_consensus.cc:2781] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:06.690218 3929 raft_consensus.cc:482] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:06.690359 3929 raft_consensus.cc:3037] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:06.692641 3929 raft_consensus.cc:504] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.693344 3929 leader_election.cc:288] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [CANDIDATE]: Term 1 election: Requested vote from peers 229fb7c6fa644267966cbd2716b51df4 (127.2.205.129:42491), 957b103a2ab84305b7124c422a1ba394 (127.2.205.131:36881)
I20260929 03:01:06.693907 3859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff53fc68e4a14ecebe0e0ff08b5b85a7" candidate_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957b103a2ab84305b7124c422a1ba394"
I20260929 03:01:06.693907 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff53fc68e4a14ecebe0e0ff08b5b85a7" candidate_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229fb7c6fa644267966cbd2716b51df4"
I20260929 03:01:06.694139 3592 raft_consensus.cc:3037] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:06.694133 3859 raft_consensus.cc:3037] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:06.697511 3859 raft_consensus.cc:2454] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbf7faa3d914318a6f8a7817f37c31c in term 1.
I20260929 03:01:06.697511 3592 raft_consensus.cc:2454] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdbf7faa3d914318a6f8a7817f37c31c in term 1.
I20260929 03:01:06.698038 3662 leader_election.cc:302] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957b103a2ab84305b7124c422a1ba394, cdbf7faa3d914318a6f8a7817f37c31c; no voters:
I20260929 03:01:06.698360 3929 raft_consensus.cc:2781] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:06.698840 3929 raft_consensus.cc:686] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [term 1 LEADER]: Becoming Leader. State: Replica: cdbf7faa3d914318a6f8a7817f37c31c, State: Running, Role: LEADER
I20260929 03:01:06.699376 3929 consensus_queue.cc:237] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } }
I20260929 03:01:06.704735 3445 catalog_manager.cc:5495] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c reported cstate change: term changed from 0 to 1, leader changed from <none> to cdbf7faa3d914318a6f8a7817f37c31c (127.2.205.130). New cstate: current_term: 1 leader_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229fb7c6fa644267966cbd2716b51df4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 42491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957b103a2ab84305b7124c422a1ba394" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdbf7faa3d914318a6f8a7817f37c31c" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39637 } health_report { overall_health: HEALTHY } } }
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:06.733628 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.718803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.2.205.129:42491)
0929 03:01:06.718984 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.718991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.719460 (+ 469us) server_negotiation.cc:407] Connection header received
0929 03:01:06.719639 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.719644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.719720 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.719840 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.721553 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.722341 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.723696 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.724134 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.724255 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.728169 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.728185 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.728202 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.728245 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.730979 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.731621 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.731630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.731632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.731769 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.732136 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.732140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.732142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.732505 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.732822 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.733097 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.733302 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 03:01:06.737991 3905 tablet.cc:2369] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 957b103a2ab84305b7124c422a1ba394: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 03:01:06.762666 3772 tablet.cc:2369] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P cdbf7faa3d914318a6f8a7817f37c31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
W20260929 03:01:06.774401 3638 tablet.cc:2369] T ff53fc68e4a14ecebe0e0ff08b5b85a7 P 229fb7c6fa644267966cbd2716b51df4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:06.776451 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.763188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.2.205.129:42491)
0929 03:01:06.763374 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.763381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.763735 (+ 354us) server_negotiation.cc:407] Connection header received
0929 03:01:06.763905 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.763911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.763979 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.764059 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.765418 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.766150 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.767411 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.767816 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.767926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.771481 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.771487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.771490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.771522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.774051 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.774604 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.774609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.774612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.774778 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.775158 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.775164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.775167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.775424 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.775651 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.775932 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.776133 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:06.819203 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.805293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.2.205.129:42491)
0929 03:01:06.805560 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.805566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.805853 (+ 287us) server_negotiation.cc:407] Connection header received
0929 03:01:06.806038 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.806042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.806110 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.806201 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.807615 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.808333 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.809649 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.810081 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.810195 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.814081 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.814086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.814090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.814129 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.816466 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.817204 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.817210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.817213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.817353 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.817728 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.817733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.817735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.818035 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.818283 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.818613 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.818844 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 03:01:06.821022 3552 tablet_service.cc:2908] Rejecting scan request for tablet ff53fc68e4a14ecebe0e0ff08b5b85a7: Uninitialized: clean time has not yet been initialized
I20260929 03:01:06.823771 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3508
I20260929 03:01:06.835861 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.824883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.2.205.129:42491)
0929 03:01:06.825080 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.825087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.825387 (+ 300us) server_negotiation.cc:407] Connection header received
0929 03:01:06.825517 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.825521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.825594 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.825672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.826777 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.827445 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.828569 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.829026 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.829124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.831218 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.831223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.831226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.831264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.833645 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.834308 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.834311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.834313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.834431 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.834713 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.834715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.834717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.834993 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.835203 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.835403 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.835580 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:06.836560 3632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:06.925158 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3508
I20260929 03:01:06.940917 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3643
Sep 29 03:01:06 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:06.954831 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:06.941684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.2.205.130:39637)
0929 03:01:06.941878 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:06.941882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:06.941902 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:06.942062 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:06.942066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:06.942143 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:06.942224 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:06.943324 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.944048 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.945315 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:06.945712 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:06.945802 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:06.949487 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:06.949502 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:06.949518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:06.949566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:06.952224 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.953118 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.953126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.953128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.953286 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:06.953605 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:06.953609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:06.953611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:06.953950 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:06.954187 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:06.954343 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:01:06.954510 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:01:06.955459 3766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:07.043028 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3643
I20260929 03:01:07.056313 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3776
Sep 29 03:01:07 dist-test-slave-530n krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:07.071136 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:07.057037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.2.205.131:36881)
0929 03:01:07.057230 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:07.057234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:07.057252 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:07.057386 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:07.057389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:07.057450 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:07.057529 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:07.058719 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:07.059523 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:07.060781 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:07.061216 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:07.061353 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:07.065501 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:07.065516 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:07.065532 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:07.065571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:07.068247 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:07.069151 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:07.069162 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:07.069165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:07.069333 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:07.069727 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:07.069731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:07.069733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:07.070132 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:07.070377 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:07.070623 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 03:01:07.070812 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:01:07.071805 3899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:07.159879 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3776
I20260929 03:01:07.172652 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3413
I20260929 03:01:07.174069 3476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:07.287614 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3413
2026-09-29T03:01:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3949 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:07 dist-test-slave-530n krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:07 dist-test-slave-530n krb5kdc[3952](info): set up 2 sockets
Sep 29 03:01:07 dist-test-slave-530n krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260929 03:01:08.272857 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:08 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:08.448551 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45109
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:08.767293 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:08.767597 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:08.767711 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:08.775450 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:08.775576 3965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:01:08.775617 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:08.775695 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:08.775763 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:08.775846 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:08.786243 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:08.787632 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:08.789019 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:08.800429 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:08.800412 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:08.800462 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:08.801442 3965 server_base.cc:1034] running on GCE node
I20260929 03:01:08.802217 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:08.803714 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:08.804972 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790650868804909 us; error 39 us; skew 500 ppm
I20260929 03:01:08.805346 3965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:08 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650868, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:08.808544 3965 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:08.810209 3965 webserver.cc:460] Webserver started at http://127.2.205.190:41897/ using document root <none> and password file <none>
I20260929 03:01:08.810988 3965 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:08.811095 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:08.811446 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:08.814790 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "c762bd326a394afd8bd63263e97b041f"
format_stamp: "Formatted at 2026-09-29 03:01:08 on dist-test-slave-530n"
server_key: "527a36d36187a14ffce5102e27ecac53"
server_key_iv: "943f077b538c30a94235cf499d361f90"
server_key_version: "encryptionkey@0"
I20260929 03:01:08.815685 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "c762bd326a394afd8bd63263e97b041f"
format_stamp: "Formatted at 2026-09-29 03:01:08 on dist-test-slave-530n"
server_key: "527a36d36187a14ffce5102e27ecac53"
server_key_iv: "943f077b538c30a94235cf499d361f90"
server_key_version: "encryptionkey@0"
I20260929 03:01:08.821406 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:01:08.825048 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:08.826481 3965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:08.826637 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "c762bd326a394afd8bd63263e97b041f"
format_stamp: "Formatted at 2026-09-29 03:01:08 on dist-test-slave-530n"
server_key: "527a36d36187a14ffce5102e27ecac53"
server_key_iv: "943f077b538c30a94235cf499d361f90"
server_key_version: "encryptionkey@0"
I20260929 03:01:08.826839 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:08.845150 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:08.848658 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:08.848888 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:08.868173 3965 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:45109
I20260929 03:01:08.868203 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:45109 every 8 connection(s)
I20260929 03:01:08.869892 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:08.873656 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:08.875428 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 3965
I20260929 03:01:08.875653 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:08.876447 2870 external_mini_cluster.cc:1428] Setting key 78501cf94bad8b65d6cf3a040dc68679
I20260929 03:01:08.885962 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Bootstrap starting.
I20260929 03:01:08.889768 4042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:08.890966 4042 log.cc:824] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:01:08 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:08.894320 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: No bootstrap required, opened a new log
I20260929 03:01:08.900197 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:08.878494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.2.205.190:45109)
0929 03:01:08.879281 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:08.879292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:08.879334 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:01:08.881051 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:08.881074 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:08.881476 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:08.881810 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:08.883532 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:08.884965 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:08.886680 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:08.887394 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:08.887545 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:08.892058 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:08.892070 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:08.892090 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:08.892139 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:08.895794 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:08.896431 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:08.896440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:08.896452 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:08.896661 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:08.897034 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:08.897039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:08.897043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:08.897766 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:08.898045 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:08.898390 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 03:01:08.898845 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":356,"threads_started":1}
I20260929 03:01:08.903891 4042 raft_consensus.cc:348] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } }
I20260929 03:01:08.904197 4042 raft_consensus.cc:374] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:08.904275 4042 raft_consensus.cc:729] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c762bd326a394afd8bd63263e97b041f, State: Initialized, Role: FOLLOWER
I20260929 03:01:08.904789 4042 consensus_queue.cc:260] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } }
I20260929 03:01:08.904999 4042 raft_consensus.cc:388] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:08.905093 4042 raft_consensus.cc:482] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:08.905208 4042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:08.907212 4042 raft_consensus.cc:504] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } }
I20260929 03:01:08.907734 4042 leader_election.cc:302] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c762bd326a394afd8bd63263e97b041f; no voters:
I20260929 03:01:08.908372 4042 leader_election.cc:288] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:08.908485 4047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:08.909127 4047 raft_consensus.cc:686] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [term 1 LEADER]: Becoming Leader. State: Replica: c762bd326a394afd8bd63263e97b041f, State: Running, Role: LEADER
I20260929 03:01:08.909607 4047 consensus_queue.cc:237] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } }
I20260929 03:01:08.910363 4042 sys_catalog.cc:564] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:08.912484 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c762bd326a394afd8bd63263e97b041f. Latest consensus state: current_term: 1 leader_uuid: "c762bd326a394afd8bd63263e97b041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } } }
I20260929 03:01:08.912962 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:08.913535 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c762bd326a394afd8bd63263e97b041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c762bd326a394afd8bd63263e97b041f" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45109 } } }
I20260929 03:01:08.913811 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:08.913920 4055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:08.920013 4055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:08.929953 4055 catalog_manager.cc:1347] Generated new cluster ID: 2f25426f8de642b48718217ae4c34aff
I20260929 03:01:08.930086 4055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:08.945703 4055 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:08.947111 4055 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:08.958303 4055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Generated new TSK 0
I20260929 03:01:08.959228 4055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:09.002705 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45109
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:09.325642 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:09.325958 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:09.326087 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:09.333671 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:09.333801 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:09.333911 4068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:09.334015 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:09.343991 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:09.345464 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:09.346855 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:09.358834 4073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:09.358860 4074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:09.358966 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:09.359606 4068 server_base.cc:1034] running on GCE node
I20260929 03:01:09.360170 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:09.361065 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:09.362300 4068 hybrid_clock.cc:648] HybridClock initialized: now 1790650869362238 us; error 37 us; skew 500 ppm
I20260929 03:01:09.362573 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:09 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:09.365609 4068 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:09.366905 4068 webserver.cc:460] Webserver started at http://127.2.205.129:40099/ using document root <none> and password file <none>
I20260929 03:01:09.367547 4068 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:09.367631 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:09.367918 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:09.370715 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "eea2234cfab2ad8221f986fff737ba38"
server_key_iv: "de1859d9c7daf5b7bb3d5bece71e9c4e"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.371454 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "eea2234cfab2ad8221f986fff737ba38"
server_key_iv: "de1859d9c7daf5b7bb3d5bece71e9c4e"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.376124 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:01:09.379621 4083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:09.381033 4068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:09.381181 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "eea2234cfab2ad8221f986fff737ba38"
server_key_iv: "de1859d9c7daf5b7bb3d5bece71e9c4e"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.381394 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:09.403394 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:09.407639 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:09.407898 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:09.410611 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:09.410704 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:09.410890 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:09.410952 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:09 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:09.436215 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.411229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:55923 (local address 127.2.205.190:45109)
0929 03:01:09.414137 (+ 2908us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:09.414142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:09.414163 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:01:09.416119 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:09.416126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:09.416246 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:09.416365 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:09.419320 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.420222 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.422707 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.423384 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.423534 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.428661 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:09.428669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:09.428676 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:09.428729 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:09.431974 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.432951 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.432959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.432962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.433153 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.433820 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.433825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.433829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.434224 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:09.434376 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:09.435535 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 03:01:09.435816 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2794,"thread_start_us":2717,"threads_started":1}
I20260929 03:01:09.438311 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.411983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.129:55923)
0929 03:01:09.412978 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:09.413035 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:09.415611 (+ 2576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:09.416895 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:09.416913 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:09.417917 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:09.419063 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.419099 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.420453 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.420460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.422492 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.422514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.423680 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.423689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.424056 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.424953 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:09.424989 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:09.428247 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:09.432151 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.432162 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.432186 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.432735 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.433314 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.433319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.433326 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.433675 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.434387 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:09.434397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:09.434989 (+ 592us) client_negotiation.cc:769] Sending connection context
0929 03:01:09.435398 (+ 409us) client_negotiation.cc:240] Negotiation successful
0929 03:01:09.436100 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":373,"threads_started":1}
I20260929 03:01:09.450281 4068 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:35233
I20260929 03:01:09.450335 4203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:35233 every 8 connection(s)
I20260929 03:01:09.451882 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:09.454094 4204 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45109
I20260929 03:01:09.454308 4204 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:09.454945 4204 heartbeater.cc:511] Master 127.2.205.190:45109 requested a full tablet report, sending...
I20260929 03:01:09.456890 3997 ts_manager.cc:194] Registered new tserver with Master: 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233)
I20260929 03:01:09.458542 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:55923
I20260929 03:01:09.459615 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.446125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:38691 (local address 127.2.205.190:45109)
0929 03:01:09.446316 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:09.446322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:09.447141 (+ 819us) server_negotiation.cc:407] Connection header received
0929 03:01:09.447325 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:09.447332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:09.447418 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:09.447868 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:09.448944 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.449998 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.451821 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.452297 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.452431 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.454582 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:09.454587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:09.454591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:09.454622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:09.457141 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.457728 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.457733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.457737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.457897 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.458225 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.458228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.458230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.458581 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:09.458713 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:09.459093 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 03:01:09.459295 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:01:09.459702 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.446264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.129:38691)
0929 03:01:09.447027 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:09.447049 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:09.447204 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:09.447589 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:09.447594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:09.448376 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:09.448795 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.448811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.450163 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.450168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.451670 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.451694 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.452516 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.452523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.452769 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.453515 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:09.453537 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:09.454420 (+ 883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:09.457265 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.457270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.457275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.457620 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.457996 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.458001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.458005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.458123 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.458699 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:09.458704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:09.458862 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 03:01:09.458979 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 03:01:09.459254 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":440,"threads_started":1}
I20260929 03:01:09.461630 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4068
I20260929 03:01:09.461838 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:09.462627 2870 external_mini_cluster.cc:1428] Setting key c4880966d09887a80bd3acd5dd1d9012
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:09.500207 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45109
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:09.816587 4208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:09.816896 4208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:09.817030 4208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:09.824263 4208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:09.824412 4208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:09.824542 4208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:09.824646 4208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:09.834954 4208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:09.836355 4208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:09.837863 4208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:09.850833 4213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:09.850943 4216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:09.850914 4214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:09.851759 4208 server_base.cc:1034] running on GCE node
I20260929 03:01:09.852298 4208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:09.853155 4208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:09.854362 4208 hybrid_clock.cc:648] HybridClock initialized: now 1790650869854295 us; error 46 us; skew 500 ppm
I20260929 03:01:09.854630 4208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:09 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:09.857506 4208 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:09.858812 4208 webserver.cc:460] Webserver started at http://127.2.205.130:44929/ using document root <none> and password file <none>
I20260929 03:01:09.859450 4208 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:09.859542 4208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:09.859841 4208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:09.862593 4208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "c38017b512ae59f928c25117e6b234c1"
server_key_iv: "2c4509efebad54a5253dcecad0e15e1d"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.863332 4208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "c38017b512ae59f928c25117e6b234c1"
server_key_iv: "2c4509efebad54a5253dcecad0e15e1d"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.868307 4208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 03:01:09.872400 4223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:09.873863 4208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:09.874006 4208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
format_stamp: "Formatted at 2026-09-29 03:01:09 on dist-test-slave-530n"
server_key: "c38017b512ae59f928c25117e6b234c1"
server_key_iv: "2c4509efebad54a5253dcecad0e15e1d"
server_key_version: "encryptionkey@0"
I20260929 03:01:09.874210 4208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:09.917323 4208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:09.921654 4208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:09.921900 4208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:09.924489 4208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:09.924662 4208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 03:01:09.924817 4208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:09.924887 4208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:09 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:09.948369 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.925399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:40373 (local address 127.2.205.190:45109)
0929 03:01:09.925589 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:09.925596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:09.927041 (+ 1445us) server_negotiation.cc:407] Connection header received
0929 03:01:09.929106 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:09.929112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:09.929201 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:09.929288 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:09.931773 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.932843 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.935817 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.936495 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.936640 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.941982 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:09.941988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:09.941992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:09.942030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:09.944519 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.945523 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.945528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.945530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.945668 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.946170 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.946176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.946179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.946573 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:09.946726 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:09.947786 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0929 03:01:09.948024 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:01:09.950008 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.926063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.130:40373)
0929 03:01:09.926863 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:09.926912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:09.928776 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:09.929588 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:09.929601 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:09.930419 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:09.931459 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.931493 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.933021 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.933027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.935615 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.935640 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.936802 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.936812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.937189 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.938116 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:09.938156 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:09.941680 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:09.944663 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.944673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.944693 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.945393 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.945813 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.945819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.945823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.946019 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.946726 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:09.946736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:09.947250 (+ 514us) client_negotiation.cc:769] Sending connection context
0929 03:01:09.947665 (+ 415us) client_negotiation.cc:240] Negotiation successful
0929 03:01:09.948315 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":340,"threads_started":1}
I20260929 03:01:09.969973 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.955902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:33387 (local address 127.2.205.190:45109)
0929 03:01:09.956243 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:09.956250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:09.957401 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 03:01:09.957556 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:09.957560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:09.957623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:09.957713 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:09.958838 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.959795 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.961486 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.962028 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.962158 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.963812 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:09.963817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:09.963820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:09.963850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:09.967182 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.967990 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.967994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.967997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.968128 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:09.968574 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:09.968579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:09.968582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:09.969012 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:09.969172 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:09.969591 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 03:01:09.969754 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260929 03:01:09.970367 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:09.955941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.130:33387)
0929 03:01:09.957278 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:09.957298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:09.957439 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:09.957781 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:09.957786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:09.958306 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:09.958704 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.958715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.959914 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.959920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.961350 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:09.961362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:09.962216 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:09.962223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:09.962392 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:09.962924 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:09.962940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:09.963623 (+ 683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:09.967312 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.967316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.967319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.967824 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.968260 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:09.968266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:09.968270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:09.968450 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:09.969143 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:09.969148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:09.969368 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 03:01:09.969526 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 03:01:09.969882 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":1024,"threads_started":1}
I20260929 03:01:09.974980 4208 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:41971
I20260929 03:01:09.975018 4342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:41971 every 8 connection(s)
I20260929 03:01:09.976087 4208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:09.976917 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4208
I20260929 03:01:09.977128 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:09.977705 2870 external_mini_cluster.cc:1428] Setting key e9aa3d9f388473d302e87b3dcc981eeb
I20260929 03:01:09.978128 4343 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45109
I20260929 03:01:09.978821 4343 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:09.979482 4343 heartbeater.cc:511] Master 127.2.205.190:45109 requested a full tablet report, sending...
I20260929 03:01:09.980948 3997 ts_manager.cc:194] Registered new tserver with Master: a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971)
I20260929 03:01:09.981787 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:40373
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:10.015605 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45109
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:10.333106 4347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:10.333371 4347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:10.333472 4347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:10.340345 4347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:10.340446 4347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:10.340536 4347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:10.340613 4347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:10.350184 4347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:45109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:10.351388 4347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:10.352778 4347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:10.365700 4353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:10.365645 4355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:10.365837 4352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:10.366278 4347 server_base.cc:1034] running on GCE node
I20260929 03:01:10.366859 4347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:10.367746 4347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:10.368944 4347 hybrid_clock.cc:648] HybridClock initialized: now 1790650870368879 us; error 38 us; skew 500 ppm
I20260929 03:01:10.369215 4347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:10 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:10.372124 4347 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:10.373483 4347 webserver.cc:460] Webserver started at http://127.2.205.131:34145/ using document root <none> and password file <none>
I20260929 03:01:10.374132 4347 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:10.374220 4347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:10.374498 4347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:10.377447 4347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "e9c000b9891a42d289f72265d6962c7d"
format_stamp: "Formatted at 2026-09-29 03:01:10 on dist-test-slave-530n"
server_key: "677e137be72948a8fd1bb128c104409b"
server_key_iv: "aef6842019d045da4ab86f20a8563195"
server_key_version: "encryptionkey@0"
I20260929 03:01:10.378193 4347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "e9c000b9891a42d289f72265d6962c7d"
format_stamp: "Formatted at 2026-09-29 03:01:10 on dist-test-slave-530n"
server_key: "677e137be72948a8fd1bb128c104409b"
server_key_iv: "aef6842019d045da4ab86f20a8563195"
server_key_version: "encryptionkey@0"
I20260929 03:01:10.383766 4347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:01:10.387375 4362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:10.388945 4347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:10.389088 4347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "e9c000b9891a42d289f72265d6962c7d"
format_stamp: "Formatted at 2026-09-29 03:01:10 on dist-test-slave-530n"
server_key: "677e137be72948a8fd1bb128c104409b"
server_key_iv: "aef6842019d045da4ab86f20a8563195"
server_key_version: "encryptionkey@0"
I20260929 03:01:10.389305 4347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:10.411908 4347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:10.416867 4347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:10.417124 4347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:10.419853 4347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:10.419965 4347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:10.420095 4347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:10.420181 4347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:10 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:10.442991 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.420383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:55533 (local address 127.2.205.190:45109)
0929 03:01:10.420565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.420572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.422233 (+ 1661us) server_negotiation.cc:407] Connection header received
0929 03:01:10.424379 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.424386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.424475 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.424560 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:10.426188 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.427099 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.430139 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.430724 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.430837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.435356 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:10.435364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:10.435369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:10.435417 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:10.438774 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.439964 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.439970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.439974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.440161 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.440811 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.440816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.440820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.441127 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:10.441273 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.442369 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.442635 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:01:10.444118 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.421205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.131:55533)
0929 03:01:10.422045 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.422085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.424095 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.424906 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.424914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.425428 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:10.426015 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.426033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.427258 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.427263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.429967 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.429987 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.430997 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.431006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.431251 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.432002 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:10.432045 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:10.435072 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:10.438977 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:10.438987 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:10.439010 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:10.439812 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:10.440283 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:10.440292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:10.440296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:10.440660 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:10.441269 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:10.441278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:10.441898 (+ 620us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.442206 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.442810 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":357,"threads_started":1}
I20260929 03:01:10.462236 4204 heartbeater.cc:503] Master 127.2.205.190:45109 was elected leader, sending a full tablet report...
I20260929 03:01:10.474162 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.450298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:53273 (local address 127.2.205.190:45109)
0929 03:01:10.450489 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.450496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.460138 (+ 9642us) server_negotiation.cc:407] Connection header received
0929 03:01:10.460300 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.460304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.460390 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.460472 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:10.462188 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.463268 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.465078 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.465635 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.465774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.467922 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:10.467929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:10.467934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:10.467972 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:10.471314 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.471964 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.471970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.471974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.472132 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.472504 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.472508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.472510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.472853 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:10.473001 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.473548 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.473835 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:01:10.474179 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.450298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.2.205.131:53273)
0929 03:01:10.459985 (+ 9687us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.460011 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.460154 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.460530 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.460535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.461320 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:10.461931 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.461954 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.463445 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.463451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.464924 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.464939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.466009 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.466014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.466142 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.466794 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:10.466812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:10.467740 (+ 928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:10.471452 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:10.471455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:10.471458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:10.471856 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:10.472238 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:10.472242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:10.472244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:10.472365 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:10.472985 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:10.472989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:10.473206 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.473365 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.473687 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9520,"thread_start_us":9384,"threads_started":1}
I20260929 03:01:10.480077 4347 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:40669
I20260929 03:01:10.480111 4481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:40669 every 8 connection(s)
I20260929 03:01:10.481395 4347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:10.482841 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4347
I20260929 03:01:10.483013 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:10.483414 2870 external_mini_cluster.cc:1428] Setting key 4d543951cd036282d7319b02eb2e6ab1
I20260929 03:01:10.483399 4482 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45109
I20260929 03:01:10.484297 4482 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:10.485107 4482 heartbeater.cc:511] Master 127.2.205.190:45109 requested a full tablet report, sending...
I20260929 03:01:10.486477 3997 ts_manager.cc:194] Registered new tserver with Master: e9c000b9891a42d289f72265d6962c7d (127.2.205.131:40669)
I20260929 03:01:10.487293 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:55533
I20260929 03:01:10.498399 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:01:10 dist-test-slave-530n krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:01:10 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:10.539816 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.526688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.2.205.190:45109)
0929 03:01:10.526952 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.526960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.527070 (+ 110us) server_negotiation.cc:407] Connection header received
0929 03:01:10.527241 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.527245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.527321 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.527415 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:10.528769 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.529570 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.530885 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.531342 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.531437 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.535080 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:10.535085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:10.535088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:10.535124 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:10.537488 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.538072 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.538075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.538077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.538222 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:10.538625 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:10.538629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:10.538632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:10.538897 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:10.539072 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.539303 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.539479 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 03:01:10.543995 3997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:10.550652 3997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:10.573195 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.564653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.2.205.129:35233)
0929 03:01:10.565878 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.565884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.565904 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:10.565974 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.565980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.566144 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.566305 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:10.566808 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.568112 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.570196 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.571994 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.572229 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.572470 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.572672 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.572888 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":966,"threads_started":1}
I20260929 03:01:10.573312 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.562750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41971 (local address 127.0.0.1:41372)
0929 03:01:10.563568 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.563649 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.563827 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.565328 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.565332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.565361 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:10.565755 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.565769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.567671 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.567677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.570393 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.570409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.572220 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.572227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.572398 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.572428 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.572495 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.572775 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":349,"thread_start_us":655,"threads_started":1}
I20260929 03:01:10.573465 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.563171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.2.205.131:40669)
0929 03:01:10.564071 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.564080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.564113 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:01:10.564212 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.564220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.564498 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.565514 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:10.565831 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.567518 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.570613 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.571832 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.572586 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.572864 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.572948 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.573130 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":1005,"threads_started":1}
I20260929 03:01:10.573335 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.564110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:35233 (local address 127.0.0.1:49576)
0929 03:01:10.565248 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.565268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.565402 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.566316 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.566321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.566345 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:10.566644 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.566653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.568274 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.568279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.570021 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.570034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.572268 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.572277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.572451 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.572480 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.572548 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.572733 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":11,"spinlock_wait_cycles":12032,"thread_start_us":980,"threads_started":1}
I20260929 03:01:10.574021 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.562779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:40669 (local address 127.0.0.1:53514)
0929 03:01:10.563568 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.564000 (+ 432us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.564094 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.564734 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.564741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.564804 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:10.565294 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.565310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.567671 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.567677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.570393 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.570409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.572033 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.572039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.572278 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.572354 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.572476 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.573761 (+ 1285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":1187,"thread_start_us":548,"threads_started":1}
I20260929 03:01:10.575423 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.563220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.2.205.130:41971)
0929 03:01:10.564713 (+ 1493us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.564721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.564772 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:01:10.564867 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.564875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.565106 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.565941 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:10.566132 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.567518 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.570570 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.571947 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.573845 (+ 1898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.574042 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.574127 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.575109 (+ 982us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":1537,"threads_started":1}
I20260929 03:01:10.577942 4134 tablet_service.cc:1467] Processing CreateTablet for tablet ea8828b32b924813bbe23af833689590 (DEFAULT_TABLE table=test-table [id=ec74e27a1ed7406da25e65d952ed13a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:10.577948 4412 tablet_service.cc:1467] Processing CreateTablet for tablet ea8828b32b924813bbe23af833689590 (DEFAULT_TABLE table=test-table [id=ec74e27a1ed7406da25e65d952ed13a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:10.579361 4134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8828b32b924813bbe23af833689590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:10.579487 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8828b32b924813bbe23af833689590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:10.579501 4273 tablet_service.cc:1467] Processing CreateTablet for tablet ea8828b32b924813bbe23af833689590 (DEFAULT_TABLE table=test-table [id=ec74e27a1ed7406da25e65d952ed13a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:10.580963 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8828b32b924813bbe23af833689590. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:10.591588 4501 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Bootstrap starting.
I20260929 03:01:10.593976 4502 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Bootstrap starting.
I20260929 03:01:10.595433 4503 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Bootstrap starting.
I20260929 03:01:10.595448 4501 tablet_bootstrap.cc:654] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:10.597748 4502 tablet_bootstrap.cc:654] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:10.597836 4501 log.cc:824] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:10.598233 4503 tablet_bootstrap.cc:654] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:10.598852 4502 log.cc:824] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:10.599754 4503 log.cc:824] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:10.601087 4501 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: No bootstrap required, opened a new log
I20260929 03:01:10.601380 4501 ts_tablet_manager.cc:1397] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 03:01:10.601799 4502 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: No bootstrap required, opened a new log
I20260929 03:01:10.602054 4502 ts_tablet_manager.cc:1397] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 03:01:10.602252 4503 tablet_bootstrap.cc:492] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: No bootstrap required, opened a new log
I20260929 03:01:10.602470 4503 ts_tablet_manager.cc:1397] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:01:10.609745 4503 raft_consensus.cc:348] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.610126 4503 raft_consensus.cc:374] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:10.610287 4503 raft_consensus.cc:729] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d59ae90ef54c2dab19e2d0fd86e6ab, State: Initialized, Role: FOLLOWER
I20260929 03:01:10.610925 4503 consensus_queue.cc:260] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.612094 4343 heartbeater.cc:503] Master 127.2.205.190:45109 was elected leader, sending a full tablet report...
I20260929 03:01:10.611980 4501 raft_consensus.cc:348] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.612442 4501 raft_consensus.cc:374] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:10.612596 4501 raft_consensus.cc:729] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dc43a2c99743f4aeabfaf4c5f9c292, State: Initialized, Role: FOLLOWER
I20260929 03:01:10.612701 4503 ts_tablet_manager.cc:1428] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 03:01:10.612741 4502 raft_consensus.cc:348] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.613178 4502 raft_consensus.cc:374] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:10.613271 4502 raft_consensus.cc:729] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Initialized, Role: FOLLOWER
I20260929 03:01:10.613374 4501 consensus_queue.cc:260] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.613837 4502 consensus_queue.cc:260] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.615212 4501 ts_tablet_manager.cc:1428] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 03:01:10.616122 4482 heartbeater.cc:503] Master 127.2.205.190:45109 was elected leader, sending a full tablet report...
I20260929 03:01:10.617101 4502 ts_tablet_manager.cc:1428] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260929 03:01:10.704604 4205 tablet.cc:2369] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:01:10.729429 4344 tablet.cc:2369] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:01:10.733071 4483 tablet.cc:2369] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:10.943979 4509 raft_consensus.cc:482] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:10.944238 4509 raft_consensus.cc:504] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.945953 4509 leader_election.cc:288] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971), 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233)
I20260929 03:01:10.951851 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.946256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:40557 (local address 127.2.205.130:41971)
0929 03:01:10.946475 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.946482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.946679 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:01:10.946871 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.946877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.946982 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.947079 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:10.947664 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.948613 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.950260 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.951002 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.951156 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.951233 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.951483 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.951605 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:01:10.951930 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.946192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41971 (local address 127.2.205.131:40557)
0929 03:01:10.946578 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.946593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.946729 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.947274 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.947279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.947308 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:10.947534 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.947543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.948726 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.948730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.950101 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.950112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.951177 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.951183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.951326 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.951350 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.951431 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.951558 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":1,"thread_start_us":216,"threads_started":1}
I20260929 03:01:10.952287 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.947064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:51227 (local address 127.2.205.129:35233)
0929 03:01:10.947256 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:10.947261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:10.947427 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:01:10.947591 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:10.947596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:10.947678 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:10.947767 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:10.948212 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.949069 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.950661 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.951406 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.951546 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.951613 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:10.951887 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:01:10.952006 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:01:10.952281 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:10.946795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:35233 (local address 127.2.205.131:51227)
0929 03:01:10.947342 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:10.947357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:10.947475 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:10.947797 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:10.947800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:10.947818 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:10.948072 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.948079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.949179 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.949184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.950526 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:10.950537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:10.951612 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:10.951618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:10.951748 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:10.951771 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:01:10.951850 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:01:10.951974 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
I20260929 03:01:10.953390 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8828b32b924813bbe23af833689590" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" is_pre_election: true
I20260929 03:01:10.953368 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8828b32b924813bbe23af833689590" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" is_pre_election: true
I20260929 03:01:10.953845 4293 raft_consensus.cc:2454] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:10.953846 4154 raft_consensus.cc:2454] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:10.954425 4363 leader_election.cc:302] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d59ae90ef54c2dab19e2d0fd86e6ab, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:10.954762 4509 raft_consensus.cc:2781] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:10.954882 4509 raft_consensus.cc:482] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:10.954978 4509 raft_consensus.cc:3037] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:10.957338 4509 raft_consensus.cc:504] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.957854 4509 leader_election.cc:288] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Requested vote from peers a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971), 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233)
I20260929 03:01:10.958359 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8828b32b924813bbe23af833689590" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
I20260929 03:01:10.958387 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8828b32b924813bbe23af833689590" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
I20260929 03:01:10.958537 4293 raft_consensus.cc:3037] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:10.958545 4154 raft_consensus.cc:3037] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:10.960618 4293 raft_consensus.cc:2454] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:10.960614 4154 raft_consensus.cc:2454] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:10.961014 4363 leader_election.cc:302] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d59ae90ef54c2dab19e2d0fd86e6ab, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:10.961282 4509 raft_consensus.cc:2781] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:10.961712 4509 raft_consensus.cc:686] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 1 LEADER]: Becoming Leader. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Running, Role: LEADER
I20260929 03:01:10.962157 4509 consensus_queue.cc:237] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } }
I20260929 03:01:10.966866 3996 catalog_manager.cc:5495] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c000b9891a42d289f72265d6962c7d (127.2.205.131). New cstate: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:11.036922 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.031854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.2.205.131:40669)
0929 03:01:11.032006 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.032010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.032156 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:01:11.032444 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.032448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.032524 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.032635 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:11.032921 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.033602 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.034917 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.035317 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.035406 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.035900 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:11.036387 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:11.036486 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.036615 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.036679 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:01:11.044406 4293 raft_consensus.cc:1261] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.044405 4154 raft_consensus.cc:1261] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.045356 4512 consensus_queue.cc:1035] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.045737 4509 consensus_queue.cc:1035] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.075326 3995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:11.075565 3995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:11.079210 3995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Sending DeleteTablet for 3 replicas of tablet ea8828b32b924813bbe23af833689590
I20260929 03:01:11.080292 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8828b32b924813bbe23af833689590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:53514
I20260929 03:01:11.080525 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8828b32b924813bbe23af833689590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:41372
I20260929 03:01:11.080518 4134 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8828b32b924813bbe23af833689590 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:49576
I20260929 03:01:11.081175 4524 tablet_replica.cc:330] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: stopping tablet replica
I20260929 03:01:11.081615 4524 raft_consensus.cc:2229] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:11.082311 4524 raft_consensus.cc:2258] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.082419 3995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:11.083178 3995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:11.084370 4524 ts_tablet_manager.cc:1905] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:11.091620 4524 ts_tablet_manager.cc:1918] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:11.091668 4272 tablet_service.cc:1467] Processing CreateTablet for tablet 2b352789dc434a71972211e5b31e2bcb (DEFAULT_TABLE table=test-table [id=cd8b69b46d1241e89aed699d559bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:11.091889 4524 log.cc:1196] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/ea8828b32b924813bbe23af833689590
I20260929 03:01:11.091889 4133 tablet_service.cc:1467] Processing CreateTablet for tablet 2b352789dc434a71972211e5b31e2bcb (DEFAULT_TABLE table=test-table [id=cd8b69b46d1241e89aed699d559bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:11.092288 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b352789dc434a71972211e5b31e2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.092377 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b352789dc434a71972211e5b31e2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.092417 4524 ts_tablet_manager.cc:1939] T ea8828b32b924813bbe23af833689590 P e9c000b9891a42d289f72265d6962c7d: Deleting consensus metadata
I20260929 03:01:11.092638 4526 tablet_replica.cc:330] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: stopping tablet replica
I20260929 03:01:11.096829 4526 raft_consensus.cc:2229] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:11.097110 4412 tablet_service.cc:1467] Processing CreateTablet for tablet 2b352789dc434a71972211e5b31e2bcb (DEFAULT_TABLE table=test-table [id=cd8b69b46d1241e89aed699d559bf914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:11.097374 4526 raft_consensus.cc:2258] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.097534 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b352789dc434a71972211e5b31e2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.098106 4525 tablet_replica.cc:330] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: stopping tablet replica
I20260929 03:01:11.098569 4525 raft_consensus.cc:2229] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:11.098966 4503 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Bootstrap starting.
I20260929 03:01:11.099033 4525 raft_consensus.cc:2258] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.100513 4503 tablet_bootstrap.cc:654] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.101473 3984 catalog_manager.cc:4841] TS e9c000b9891a42d289f72265d6962c7d (127.2.205.131:40669): tablet ea8828b32b924813bbe23af833689590 (table test-table [id=ec74e27a1ed7406da25e65d952ed13a0]) successfully deleted
I20260929 03:01:11.101789 4501 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Bootstrap starting.
I20260929 03:01:11.103358 4501 tablet_bootstrap.cc:654] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.106459 4526 ts_tablet_manager.cc:1905] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:11.107160 4502 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Bootstrap starting.
I20260929 03:01:11.108858 4502 tablet_bootstrap.cc:654] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.109462 4525 ts_tablet_manager.cc:1905] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:11.124500 4503 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: No bootstrap required, opened a new log
I20260929 03:01:11.124799 4503 ts_tablet_manager.cc:1397] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20260929 03:01:11.125748 4503 raft_consensus.cc:348] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.126109 4503 raft_consensus.cc:374] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.126266 4503 raft_consensus.cc:729] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d59ae90ef54c2dab19e2d0fd86e6ab, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.126616 4503 consensus_queue.cc:260] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.128206 4503 ts_tablet_manager.cc:1428] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:01:11.131095 4526 ts_tablet_manager.cc:1918] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:11.132719 4501 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: No bootstrap required, opened a new log
I20260929 03:01:11.132983 4501 ts_tablet_manager.cc:1397] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.017s
I20260929 03:01:11.133190 4526 log.cc:1196] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/ea8828b32b924813bbe23af833689590
I20260929 03:01:11.133328 4525 ts_tablet_manager.cc:1918] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:11.133494 4502 tablet_bootstrap.cc:492] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: No bootstrap required, opened a new log
I20260929 03:01:11.133562 4525 log.cc:1196] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/ea8828b32b924813bbe23af833689590
I20260929 03:01:11.133706 4502 ts_tablet_manager.cc:1397] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.008s
I20260929 03:01:11.134191 4525 ts_tablet_manager.cc:1939] T ea8828b32b924813bbe23af833689590 P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting consensus metadata
I20260929 03:01:11.134125 4501 raft_consensus.cc:348] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.134356 4501 raft_consensus.cc:374] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.134462 4501 raft_consensus.cc:729] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dc43a2c99743f4aeabfaf4c5f9c292, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.134615 4502 raft_consensus.cc:348] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.134727 4501 consensus_queue.cc:260] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.134938 4502 raft_consensus.cc:374] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.135078 4502 raft_consensus.cc:729] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.135267 4501 ts_tablet_manager.cc:1428] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 03:01:11.135468 4502 consensus_queue.cc:260] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.135922 4526 ts_tablet_manager.cc:1939] T ea8828b32b924813bbe23af833689590 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting consensus metadata
I20260929 03:01:11.137949 4502 ts_tablet_manager.cc:1428] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 03:01:11.137928 3981 catalog_manager.cc:4841] TS a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971): tablet ea8828b32b924813bbe23af833689590 (table test-table [id=ec74e27a1ed7406da25e65d952ed13a0]) successfully deleted
I20260929 03:01:11.139351 3981 catalog_manager.cc:4841] TS 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233): tablet ea8828b32b924813bbe23af833689590 (table test-table [id=ec74e27a1ed7406da25e65d952ed13a0]) successfully deleted
I20260929 03:01:11.457497 4509 raft_consensus.cc:482] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:11.457732 4509 raft_consensus.cc:504] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.458443 4509 leader_election.cc:288] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233), a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971)
I20260929 03:01:11.459057 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b352789dc434a71972211e5b31e2bcb" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" is_pre_election: true
I20260929 03:01:11.459071 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b352789dc434a71972211e5b31e2bcb" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" is_pre_election: true
I20260929 03:01:11.459311 4293 raft_consensus.cc:2454] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:11.459302 4154 raft_consensus.cc:2454] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:11.459769 4363 leader_election.cc:302] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d59ae90ef54c2dab19e2d0fd86e6ab, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:11.460038 4509 raft_consensus.cc:2781] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:11.460155 4509 raft_consensus.cc:482] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:11.460228 4509 raft_consensus.cc:3037] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.463148 4509 raft_consensus.cc:504] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.463773 4509 leader_election.cc:288] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Requested vote from peers 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233), a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971)
I20260929 03:01:11.464311 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b352789dc434a71972211e5b31e2bcb" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
I20260929 03:01:11.464515 4154 raft_consensus.cc:3037] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.464468 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b352789dc434a71972211e5b31e2bcb" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
I20260929 03:01:11.464656 4293 raft_consensus.cc:3037] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.467420 4154 raft_consensus.cc:2454] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:11.467535 4293 raft_consensus.cc:2454] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:11.467891 4363 leader_election.cc:302] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d59ae90ef54c2dab19e2d0fd86e6ab, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:11.468205 4509 raft_consensus.cc:2781] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:11.468364 4509 raft_consensus.cc:686] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 1 LEADER]: Becoming Leader. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Running, Role: LEADER
I20260929 03:01:11.468665 4509 consensus_queue.cc:237] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.471501 3996 catalog_manager.cc:5495] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c000b9891a42d289f72265d6962c7d (127.2.205.131). New cstate: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } health_report { overall_health: HEALTHY } } }
Sep 29 03:01:11 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650868, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:11.525902 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.506945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45109 (local address 127.0.0.1:56982)
0929 03:01:11.507611 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:11.507627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:11.507756 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:11.508045 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:11.508051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:11.508642 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:11.508910 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.508919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.509918 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.509921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.510887 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.510898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.511570 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.511576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.511719 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.518317 (+ 6598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:11.518336 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:11.520484 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:11.523442 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:11.523455 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:11.523463 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:11.523888 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:11.524214 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:11.524219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:11.524222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:11.524379 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:11.524894 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:11.524902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:11.525058 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:01:11.525204 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 03:01:11.525464 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":316,"threads_started":1}
I20260929 03:01:11.525939 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.506931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.2.205.190:45109)
0929 03:01:11.507489 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.507493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.507695 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:01:11.507854 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.507857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.507919 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.508012 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.509046 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.509802 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.510990 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.511423 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.511510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.520682 (+ 9172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.520687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.520690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.520719 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.523301 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.523986 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.523992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.523995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.524103 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.524471 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.524477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.524480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.524743 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.524898 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.525299 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.525591 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
I20260929 03:01:11.530623 3996 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:56982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:01:11.531564 3996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:11.537670 4133 tablet_service.cc:1467] Processing CreateTablet for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78c6a9d785114bedb47647d5db37baa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:11.537732 4273 tablet_service.cc:1467] Processing CreateTablet for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78c6a9d785114bedb47647d5db37baa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:11.538204 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.538221 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.538174 4412 tablet_service.cc:1467] Processing CreateTablet for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=78c6a9d785114bedb47647d5db37baa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:11.538724 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294ca19a8b1d4c9195b3ad49dd0a1ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:11.544241 4502 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: Bootstrap starting.
I20260929 03:01:11.545357 4503 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Bootstrap starting.
I20260929 03:01:11.545781 4502 tablet_bootstrap.cc:654] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.546751 4503 tablet_bootstrap.cc:654] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.547832 4502 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: No bootstrap required, opened a new log
I20260929 03:01:11.547953 4501 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: Bootstrap starting.
I20260929 03:01:11.548048 4502 ts_tablet_manager.cc:1397] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 03:01:11.548633 4502 raft_consensus.cc:348] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.548892 4502 raft_consensus.cc:374] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.548969 4502 raft_consensus.cc:729] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.549191 4502 consensus_queue.cc:260] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.549312 4501 tablet_bootstrap.cc:654] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:11.549666 4502 ts_tablet_manager.cc:1428] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:01:11.549582 4509 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.549945 4509 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.550669 4503 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: No bootstrap required, opened a new log
I20260929 03:01:11.550823 4503 ts_tablet_manager.cc:1397] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:01:11.551484 4503 raft_consensus.cc:348] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.551720 4503 raft_consensus.cc:374] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.551841 4503 raft_consensus.cc:729] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d59ae90ef54c2dab19e2d0fd86e6ab, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.552116 4503 consensus_queue.cc:260] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.552724 4503 ts_tablet_manager.cc:1428] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:01:11.552634 4507 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.553069 4507 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.554582 4501 tablet_bootstrap.cc:492] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: No bootstrap required, opened a new log
I20260929 03:01:11.554741 4501 ts_tablet_manager.cc:1397] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 03:01:11.555384 4501 raft_consensus.cc:348] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.555608 4501 raft_consensus.cc:374] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:11.555681 4501 raft_consensus.cc:729] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dc43a2c99743f4aeabfaf4c5f9c292, State: Initialized, Role: FOLLOWER
I20260929 03:01:11.555908 4501 consensus_queue.cc:260] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.556710 4501 ts_tablet_manager.cc:1428] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:01:11.556694 4508 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.557097 4508 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.615643 4509 raft_consensus.cc:482] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:11.615850 4509 raft_consensus.cc:504] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.616437 4509 leader_election.cc:288] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233), a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971)
I20260929 03:01:11.616954 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294ca19a8b1d4c9195b3ad49dd0a1ef6" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" is_pre_election: true
I20260929 03:01:11.617048 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294ca19a8b1d4c9195b3ad49dd0a1ef6" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" is_pre_election: true
I20260929 03:01:11.617249 4154 raft_consensus.cc:2454] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:11.617285 4293 raft_consensus.cc:2454] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 0.
I20260929 03:01:11.617657 4363 leader_election.cc:302] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dc43a2c99743f4aeabfaf4c5f9c292, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:11.617926 4509 raft_consensus.cc:2781] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:11.618029 4509 raft_consensus.cc:482] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:11.618125 4509 raft_consensus.cc:3037] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.619971 4509 raft_consensus.cc:504] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.620445 4509 leader_election.cc:288] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Requested vote from peers 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233), a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971)
I20260929 03:01:11.620875 4154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294ca19a8b1d4c9195b3ad49dd0a1ef6" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292"
I20260929 03:01:11.620959 4293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294ca19a8b1d4c9195b3ad49dd0a1ef6" candidate_uuid: "e9c000b9891a42d289f72265d6962c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab"
I20260929 03:01:11.621107 4154 raft_consensus.cc:3037] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.621160 4293 raft_consensus.cc:3037] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:11.623862 4154 raft_consensus.cc:2454] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:11.623862 4293 raft_consensus.cc:2454] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c000b9891a42d289f72265d6962c7d in term 1.
I20260929 03:01:11.624214 4363 leader_election.cc:302] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62dc43a2c99743f4aeabfaf4c5f9c292, e9c000b9891a42d289f72265d6962c7d; no voters:
I20260929 03:01:11.624446 4509 raft_consensus.cc:2781] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:11.624606 4509 raft_consensus.cc:686] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [term 1 LEADER]: Becoming Leader. State: Replica: e9c000b9891a42d289f72265d6962c7d, State: Running, Role: LEADER
I20260929 03:01:11.624886 4509 consensus_queue.cc:237] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } }
I20260929 03:01:11.626339 4512 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: TxnStatusTablet state changed. Reason: New leader e9c000b9891a42d289f72265d6962c7d. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.626595 4512 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:11.627099 4542 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:01:11.627293 4542 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:01:11.627502 3995 catalog_manager.cc:5495] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c000b9891a42d289f72265d6962c7d (127.2.205.131). New cstate: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } health_report { overall_health: HEALTHY } } }
I20260929 03:01:11.648257 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.643264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:40669 (local address 127.0.0.1:34244)
0929 03:01:11.643484 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:11.643504 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:11.643636 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:11.644113 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:11.644118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:11.644129 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:11.644303 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.644312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.645365 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.645371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.646327 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.646338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.647075 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.647081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.647259 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.647310 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:11.647711 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:11.647728 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:01:11.647795 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:01:11.647941 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:01:11.648281 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.643313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.2.205.131:40669)
0929 03:01:11.643862 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.643866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.643880 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:01:11.643927 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.643930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.643991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.644071 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:11.644456 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.645240 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.646453 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.646943 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.647055 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.647485 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:11.647580 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:11.647653 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.647830 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.647902 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":280,"threads_started":1}
I20260929 03:01:11.652282 4154 raft_consensus.cc:1261] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.652285 4293 raft_consensus.cc:1261] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.652866 4509 consensus_queue.cc:1035] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.653365 4512 consensus_queue.cc:1035] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.655735 4507 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: TxnStatusTablet state changed. Reason: New leader e9c000b9891a42d289f72265d6962c7d. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.656023 4507 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.657153 4546 mvcc.cc:205] Tried to move back new op lower bound from 7334505970281582592 to 7334505970177478656. Current Snapshot: MvccSnapshot[applied={T|T < 7334505970281582592}]
I20260929 03:01:11.659082 4547 mvcc.cc:205] Tried to move back new op lower bound from 7334505970281582592 to 7334505970177478656. Current Snapshot: MvccSnapshot[applied={T|T < 7334505970281582592}]
I20260929 03:01:11.663270 4507 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.663475 4507 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P a1d59ae90ef54c2dab19e2d0fd86e6ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.666883 4512 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.667136 4512 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:11.672017 4508 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: TxnStatusTablet state changed. Reason: New leader e9c000b9891a42d289f72265d6962c7d. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.672261 4508 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.672505 4509 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.672797 4509 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P e9c000b9891a42d289f72265d6962c7d: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:11.673934 4549 mvcc.cc:205] Tried to move back new op lower bound from 7334505970281582592 to 7334505970177478656. Current Snapshot: MvccSnapshot[applied={T|T < 7334505970281582592}]
I20260929 03:01:11.685322 4508 tablet_replica.cc:439] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9c000b9891a42d289f72265d6962c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 } } peers { permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 } } peers { permanent_uuid: "e9c000b9891a42d289f72265d6962c7d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40669 } } }
I20260929 03:01:11.685606 4508 tablet_replica.cc:442] T 294ca19a8b1d4c9195b3ad49dd0a1ef6 P 62dc43a2c99743f4aeabfaf4c5f9c292: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:11.691630 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.684174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:39367 (local address 127.2.205.131:40669)
0929 03:01:11.684345 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.684351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.685038 (+ 687us) server_negotiation.cc:407] Connection header received
0929 03:01:11.686274 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.686279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.686365 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.686455 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:11.687046 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.687899 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.689955 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.690401 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.690506 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.690996 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:11.691093 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:11.691155 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.691293 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.691390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":52}
I20260929 03:01:11.691953 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.684281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:40669 (local address 127.2.205.131:39367)
0929 03:01:11.684949 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:11.684967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:11.686121 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:11.686476 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:11.686482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:11.686496 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:11.686889 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.686902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.688032 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.688038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.689795 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:11.689809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.690564 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.690571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:11.690771 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.690846 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:11.691184 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:11.691195 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:01:11.691277 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:01:11.691460 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":20,"thread_start_us":338,"threads_started":1}
I20260929 03:01:11.700551 4293 raft_consensus.cc:1261] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.700664 4153 raft_consensus.cc:1261] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Refusing update from remote peer e9c000b9891a42d289f72265d6962c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:11.701115 4509 consensus_queue.cc:1035] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d59ae90ef54c2dab19e2d0fd86e6ab" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.701372 4512 consensus_queue.cc:1035] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dc43a2c99743f4aeabfaf4c5f9c292" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:11.752938 3995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:11.753187 3995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:11.756206 3995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c762bd326a394afd8bd63263e97b041f: Sending DeleteTablet for 3 replicas of tablet 2b352789dc434a71972211e5b31e2bcb
I20260929 03:01:11.757017 4133 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b352789dc434a71972211e5b31e2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:49576
I20260929 03:01:11.757054 4273 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b352789dc434a71972211e5b31e2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:41372
I20260929 03:01:11.757263 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b352789dc434a71972211e5b31e2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:53514
I20260929 03:01:11.769899 4562 tablet_replica.cc:330] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: stopping tablet replica
I20260929 03:01:11.770243 4562 raft_consensus.cc:2229] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:11.770565 4562 raft_consensus.cc:2258] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.771687 4562 ts_tablet_manager.cc:1905] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:11.779048 4563 tablet_replica.cc:330] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: stopping tablet replica
I20260929 03:01:11.779333 4563 raft_consensus.cc:2229] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:11.779731 4563 raft_consensus.cc:2258] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.781164 4563 ts_tablet_manager.cc:1905] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:11.788725 4563 ts_tablet_manager.cc:1918] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:11.789198 4562 ts_tablet_manager.cc:1918] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:11.789386 4562 log.cc:1196] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/2b352789dc434a71972211e5b31e2bcb
I20260929 03:01:11.789894 4562 ts_tablet_manager.cc:1939] T 2b352789dc434a71972211e5b31e2bcb P a1d59ae90ef54c2dab19e2d0fd86e6ab: Deleting consensus metadata
I20260929 03:01:11.791114 3981 catalog_manager.cc:4841] TS a1d59ae90ef54c2dab19e2d0fd86e6ab (127.2.205.130:41971): tablet 2b352789dc434a71972211e5b31e2bcb (table test-table [id=cd8b69b46d1241e89aed699d559bf914]) successfully deleted
I20260929 03:01:11.791589 4563 log.cc:1196] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/2b352789dc434a71972211e5b31e2bcb
I20260929 03:01:11.792158 4563 ts_tablet_manager.cc:1939] T 2b352789dc434a71972211e5b31e2bcb P 62dc43a2c99743f4aeabfaf4c5f9c292: Deleting consensus metadata
I20260929 03:01:11.793237 3981 catalog_manager.cc:4841] TS 62dc43a2c99743f4aeabfaf4c5f9c292 (127.2.205.129:35233): tablet 2b352789dc434a71972211e5b31e2bcb (table test-table [id=cd8b69b46d1241e89aed699d559bf914]) successfully deleted
I20260929 03:01:11.793350 4564 tablet_replica.cc:330] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: stopping tablet replica
I20260929 03:01:11.793545 4564 raft_consensus.cc:2229] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:11.793987 4564 raft_consensus.cc:2258] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:11.794971 4564 ts_tablet_manager.cc:1905] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 03:01:11 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:11.802686 4564 ts_tablet_manager.cc:1918] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:11.802834 4564 log.cc:1196] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/2b352789dc434a71972211e5b31e2bcb
I20260929 03:01:11.803329 4564 ts_tablet_manager.cc:1939] T 2b352789dc434a71972211e5b31e2bcb P e9c000b9891a42d289f72265d6962c7d: Deleting consensus metadata
I20260929 03:01:11.804425 3984 catalog_manager.cc:4841] TS e9c000b9891a42d289f72265d6962c7d (127.2.205.131:40669): tablet 2b352789dc434a71972211e5b31e2bcb (table test-table [id=cd8b69b46d1241e89aed699d559bf914]) successfully deleted
I20260929 03:01:11.807298 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.779825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.2.205.129:35233)
0929 03:01:11.790893 (+ 11068us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.790900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.791806 (+ 906us) server_negotiation.cc:407] Connection header received
0929 03:01:11.791877 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.791883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.791970 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.792069 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.795301 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.796088 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.797351 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.797777 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.797881 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.801591 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.801604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.801618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.801665 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.804620 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.805380 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.805387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.805390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.805518 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.805869 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.805873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.805875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.806210 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.806486 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.806776 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.806983 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10942,"thread_start_us":11879,"threads_started":1}
W20260929 03:01:11.807907 4199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60300
I20260929 03:01:11.822757 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.810585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.2.205.190:45109)
0929 03:01:11.810818 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.810822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.810972 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:01:11.811133 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.811137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.811209 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.811303 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.812485 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.813231 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.814392 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.815081 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.815206 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.817899 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.817904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.817907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.817938 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.820302 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.821076 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.821080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.821082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.821191 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.821573 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.821577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.821579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.821841 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.822009 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.822361 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.822519 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260929 03:01:11.823298 3995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56992
I20260929 03:01:11.825155 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4068
I20260929 03:01:11.841725 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.825941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.2.205.129:35233)
0929 03:01:11.826159 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.826165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.829907 (+ 3742us) server_negotiation.cc:407] Connection header received
0929 03:01:11.830028 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.830034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.830113 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.830209 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.831278 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.832333 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.833490 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.834141 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.834266 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.836155 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.836161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.836166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.836204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.839315 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.839895 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.839900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.839904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.840041 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.840325 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.840330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.840334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.840642 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.840943 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.841158 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.841374 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 03:01:11.842381 4199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60302
W20260929 03:01:11.842918 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4068: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:11.843109 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4068
I20260929 03:01:11.862370 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4208
Sep 29 03:01:11 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:11.880385 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.863119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.2.205.130:41971)
0929 03:01:11.863605 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.863609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.863625 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:01:11.863675 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.863678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.863737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.863823 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.865305 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.866086 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.867805 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.868455 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.868571 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.873522 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.873543 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.873563 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.873608 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.877478 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.878180 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.878190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.878194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.878369 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.878729 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.878734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.878737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.879178 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.879529 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.879781 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.880011 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
W20260929 03:01:11.881107 4338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45028
W20260929 03:01:11.881770 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:11.881950 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4208
I20260929 03:01:11.902259 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4347
Sep 29 03:01:11 dist-test-slave-530n krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:11.917865 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:11.903154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.2.205.131:40669)
0929 03:01:11.903375 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:11.903381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:11.903399 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:11.903448 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:11.903454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:11.903529 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:11.903634 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:11.904841 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.905652 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.906854 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:11.907304 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:11.907418 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:11.911741 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:11.911759 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:11.911779 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:11.911819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:11.914915 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.915650 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.915656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.915659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.915805 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:11.916197 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:11.916200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:11.916202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:11.916573 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:11.916868 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:11.917400 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 03:01:11.917557 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 03:01:11.918540 4477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34256
W20260929 03:01:11.919188 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:11.919353 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4347
I20260929 03:01:11.936914 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 3965
I20260929 03:01:11.937968 4037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:12.049844 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 3965
2026-09-29T03:01:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4762 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:12 dist-test-slave-530n krb5kdc[4575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:12 dist-test-slave-530n krb5kdc[4575](info): set up 2 sockets
Sep 29 03:01:12 dist-test-slave-530n krb5kdc[4575](info): commencing operation
krb5kdc: starting...
W20260929 03:01:13.037014 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:13 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:13.201905 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35991
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:13.508102 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:13.508383 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:13.508498 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:13.515667 4588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:13.515758 4588 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:01:13.515820 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:13.515902 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:13.515960 4588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:13.516039 4588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:13.525694 4588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35991
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:13.526939 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:13.528285 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:13.539238 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:13.539279 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:13.539300 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:13.540261 4588 server_base.cc:1034] running on GCE node
I20260929 03:01:13.541157 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:13.542483 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:13.543720 4588 hybrid_clock.cc:648] HybridClock initialized: now 1790650873543645 us; error 39 us; skew 500 ppm
I20260929 03:01:13.544023 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:13 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650873, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:13.546909 4588 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:13.548221 4588 webserver.cc:460] Webserver started at http://127.2.205.190:39209/ using document root <none> and password file <none>
I20260929 03:01:13.548913 4588 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:13.549007 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:13.549301 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:13.552098 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "04409471a2264ec780a7cae16bcd8528"
format_stamp: "Formatted at 2026-09-29 03:01:13 on dist-test-slave-530n"
server_key: "f2a8ecad4732cc88d75e147819fc91cc"
server_key_iv: "5cda69bb12f893020ee33530fc63124a"
server_key_version: "encryptionkey@0"
I20260929 03:01:13.552891 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "04409471a2264ec780a7cae16bcd8528"
format_stamp: "Formatted at 2026-09-29 03:01:13 on dist-test-slave-530n"
server_key: "f2a8ecad4732cc88d75e147819fc91cc"
server_key_iv: "5cda69bb12f893020ee33530fc63124a"
server_key_version: "encryptionkey@0"
I20260929 03:01:13.557710 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 03:01:13.561362 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:13.562628 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:13.562825 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "04409471a2264ec780a7cae16bcd8528"
format_stamp: "Formatted at 2026-09-29 03:01:13 on dist-test-slave-530n"
server_key: "f2a8ecad4732cc88d75e147819fc91cc"
server_key_iv: "5cda69bb12f893020ee33530fc63124a"
server_key_version: "encryptionkey@0"
I20260929 03:01:13.563004 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:13.579455 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:13.583400 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:13.583642 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:13.603247 4588 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35991
I20260929 03:01:13.603297 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35991 every 8 connection(s)
I20260929 03:01:13.604514 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:13.606978 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4588
I20260929 03:01:13.607174 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:13.607930 2870 external_mini_cluster.cc:1428] Setting key d882c6876d18e6a2fd743e5233d6bbe6
I20260929 03:01:13.608873 4665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:13.620093 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Bootstrap starting.
Sep 29 03:01:13 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:13.623591 4665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:13.624866 4665 log.cc:824] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:13.628178 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: No bootstrap required, opened a new log
I20260929 03:01:13.629666 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:13.609899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.2.205.190:35991)
0929 03:01:13.610625 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:13.610636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:13.610676 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:01:13.612142 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:13.612163 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:13.612546 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:13.612934 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:13.614311 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:13.615604 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:13.616852 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:13.617550 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:13.617721 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:13.621540 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:13.621550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:13.621568 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:13.621616 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:13.625324 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:13.626165 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:13.626172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:13.626178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:13.626368 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:13.626708 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:13.626713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:13.626716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:13.627358 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:13.627681 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:13.628003 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 03:01:13.628389 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":339,"threads_started":1}
I20260929 03:01:13.635548 4665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } }
I20260929 03:01:13.635852 4665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:13.635919 4665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04409471a2264ec780a7cae16bcd8528, State: Initialized, Role: FOLLOWER
I20260929 03:01:13.636361 4665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } }
I20260929 03:01:13.636534 4665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:13.636620 4665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:13.636732 4665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:13.638705 4665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } }
I20260929 03:01:13.639101 4665 leader_election.cc:302] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04409471a2264ec780a7cae16bcd8528; no voters:
I20260929 03:01:13.639571 4665 leader_election.cc:288] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:13.639668 4670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:13.640259 4670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [term 1 LEADER]: Becoming Leader. State: Replica: 04409471a2264ec780a7cae16bcd8528, State: Running, Role: LEADER
I20260929 03:01:13.640735 4670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } }
I20260929 03:01:13.641430 4665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:13.643023 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04409471a2264ec780a7cae16bcd8528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } } }
I20260929 03:01:13.643385 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:13.644115 4675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:13.644012 4672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04409471a2264ec780a7cae16bcd8528. Latest consensus state: current_term: 1 leader_uuid: "04409471a2264ec780a7cae16bcd8528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04409471a2264ec780a7cae16bcd8528" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35991 } } }
I20260929 03:01:13.645135 4672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:13.650357 4675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:13.659075 4675 catalog_manager.cc:1347] Generated new cluster ID: 675d6769ea614c5596d8da13a23718bd
I20260929 03:01:13.659180 4675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:13.676381 4675 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:13.677665 4675 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:13.685899 4675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Generated new TSK 0
I20260929 03:01:13.686666 4675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:13.734706 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35991
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:14.059554 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:14.059839 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:14.059968 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:14.067025 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:14.067137 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:14.067224 4691 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:01:14.067279 4691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:14.067344 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:14.076658 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:14.077867 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:14.079051 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:14.091120 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:14.091181 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:14.091158 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:14.091835 4691 server_base.cc:1034] running on GCE node
I20260929 03:01:14.092406 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:14.093405 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:14.094604 4691 hybrid_clock.cc:648] HybridClock initialized: now 1790650874094546 us; error 47 us; skew 500 ppm
I20260929 03:01:14.094908 4691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:14 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:14.097843 4691 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:14.099227 4691 webserver.cc:460] Webserver started at http://127.2.205.129:37953/ using document root <none> and password file <none>
I20260929 03:01:14.099866 4691 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:14.099979 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:14.100265 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:14.103084 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "cce0b0cfec7642b99873fcdd433553c4"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "df62dc44fe04df9ca5b8b7812d105790"
server_key_iv: "b155d4b40e573d07fc0717ac2b81c9f5"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.103788 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "cce0b0cfec7642b99873fcdd433553c4"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "df62dc44fe04df9ca5b8b7812d105790"
server_key_iv: "b155d4b40e573d07fc0717ac2b81c9f5"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.108737 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 03:01:14.112107 4706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:14.113583 4691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:14.113752 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "cce0b0cfec7642b99873fcdd433553c4"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "df62dc44fe04df9ca5b8b7812d105790"
server_key_iv: "b155d4b40e573d07fc0717ac2b81c9f5"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.113940 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:14.146576 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:14.150516 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:14.150763 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:14.153332 4691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:14.153447 4691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:14.153548 4691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:14.153633 4691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:14 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:14.175283 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.153927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:35017 (local address 127.2.205.190:35991)
0929 03:01:14.154363 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:14.154370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:14.155578 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 03:01:14.157549 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:14.157554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:14.157643 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:14.157736 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:14.159694 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.160614 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.163024 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.163665 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.163806 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.168616 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:14.168622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:14.168627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:14.168668 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:14.171699 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.172642 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.172647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.172651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.172841 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.173346 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.173352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.173355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.173706 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:14.173861 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:14.174708 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 03:01:14.174926 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":201,"threads_started":1}
I20260929 03:01:14.176666 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.154675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.129:35017)
0929 03:01:14.155435 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:14.155471 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:14.157288 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:14.157997 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:14.158005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:14.158672 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:14.159535 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.159555 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.160770 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.160779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.162873 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.162889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.163921 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.163933 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.164225 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.164964 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:14.165001 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:14.168340 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:14.171846 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.171856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.171879 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.172500 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.172962 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.172967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.172971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.173185 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.173947 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:14.173955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:14.174357 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 03:01:14.174611 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 03:01:14.175146 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":343,"threads_started":1}
I20260929 03:01:14.191486 4691 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:44633
I20260929 03:01:14.191557 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:44633 every 8 connection(s)
I20260929 03:01:14.192875 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:14.194878 4827 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35991
I20260929 03:01:14.195115 4827 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:14.195879 4827 heartbeater.cc:511] Master 127.2.205.190:35991 requested a full tablet report, sending...
I20260929 03:01:14.197170 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.183115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:51821 (local address 127.2.205.190:35991)
0929 03:01:14.183298 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:14.183305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:14.184017 (+ 712us) server_negotiation.cc:407] Connection header received
0929 03:01:14.184172 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:14.184176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:14.184232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:14.184348 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:14.185524 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.186550 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.188853 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.189260 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.189358 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.191579 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:14.191586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:14.191591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:14.191628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:14.194440 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.195323 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.195327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.195329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.195448 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.195825 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.195828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.195830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.196094 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:14.196205 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:14.196629 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 03:01:14.196883 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:01:14.197208 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.183170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.129:51821)
0929 03:01:14.183913 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:14.183934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:14.184076 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:14.184372 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:14.184380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:14.184970 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:14.185388 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.185398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.186721 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.186727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.188669 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.188685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.189445 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.189452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.189637 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.190210 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:14.190237 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:14.191372 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:14.194667 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.194672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.194675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.195213 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.195550 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.195554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.195557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.195708 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.196197 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:14.196201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:14.196394 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 03:01:14.196544 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 03:01:14.196851 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":401,"threads_started":1}
I20260929 03:01:14.198042 4620 ts_manager.cc:194] Registered new tserver with Master: cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633)
I20260929 03:01:14.199601 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:35017
I20260929 03:01:14.202375 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4691
I20260929 03:01:14.202528 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:14.203051 2870 external_mini_cluster.cc:1428] Setting key f548f66ed42ef5b68f929dab073a7dba
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:14.237936 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35991
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:14.541765 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:14.542078 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:14.542238 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:14.549656 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:14.549793 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:14.549914 4831 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:01:14.549979 4831 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:14.550081 4831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:14.559767 4831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:14.561067 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:14.562381 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:14.576156 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:14.576159 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:14.576223 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:14.577056 4831 server_base.cc:1034] running on GCE node
I20260929 03:01:14.577690 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:14.578733 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:14.579933 4831 hybrid_clock.cc:648] HybridClock initialized: now 1790650874579884 us; error 46 us; skew 500 ppm
I20260929 03:01:14.580261 4831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:14 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:14.583433 4831 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:14.584995 4831 webserver.cc:460] Webserver started at http://127.2.205.130:38493/ using document root <none> and password file <none>
I20260929 03:01:14.585662 4831 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:14.585764 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:14.586030 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:14.589074 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "9364ca482ac640dba903f06dbe54ee73"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "4f4cd351c7efa6a0de2d1b8c58c42d12"
server_key_iv: "706c80f19617f756ef8eb00f0619e1dd"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.589802 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "9364ca482ac640dba903f06dbe54ee73"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "4f4cd351c7efa6a0de2d1b8c58c42d12"
server_key_iv: "706c80f19617f756ef8eb00f0619e1dd"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.595009 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:14.598618 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:14.600082 4831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:14.600279 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "9364ca482ac640dba903f06dbe54ee73"
format_stamp: "Formatted at 2026-09-29 03:01:14 on dist-test-slave-530n"
server_key: "4f4cd351c7efa6a0de2d1b8c58c42d12"
server_key_iv: "706c80f19617f756ef8eb00f0619e1dd"
server_key_version: "encryptionkey@0"
I20260929 03:01:14.600473 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:14.624323 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:14.628648 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:14.628942 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:14.631726 4831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:14.631837 4831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:14.631951 4831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:14.632022 4831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:14 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:14.654932 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.632409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:39279 (local address 127.2.205.190:35991)
0929 03:01:14.632608 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:14.632615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:14.634183 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 03:01:14.636372 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:14.636379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:14.636509 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:14.636632 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:14.638899 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.639905 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.642314 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.642910 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.643062 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.648301 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:14.648309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:14.648314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:14.648361 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:14.651579 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.652256 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.652261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.652266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.652409 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.652829 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.652835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.652839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.653229 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:14.653372 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:14.654357 (+ 985us) server_negotiation.cc:299] Negotiation successful
0929 03:01:14.654580 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:01:14.655970 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.633120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.130:39279)
0929 03:01:14.633990 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:14.634039 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:14.635990 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:14.637002 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:14.637015 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:14.637786 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:14.638711 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.638738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.640082 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.640088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.642123 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.642145 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.643410 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.643416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.643729 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.644571 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:14.644615 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:14.647982 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:14.651723 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.651731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.651744 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.652142 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.652508 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.652511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.652513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.652711 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.653383 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:14.653395 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:14.653885 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 03:01:14.654256 (+ 371us) client_negotiation.cc:240] Negotiation successful
0929 03:01:14.654820 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":345,"threads_started":1}
I20260929 03:01:14.675695 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.661502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.130:47363)
0929 03:01:14.662134 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:14.662162 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:14.662282 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:14.662660 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:14.662664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:14.663097 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:14.663424 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.663439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.664987 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.664992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.666694 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:14.666709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.667694 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.667702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:14.667897 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.668559 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:14.668584 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:14.669936 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:14.672864 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.672870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.672875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.673497 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.673903 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:14.673912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:14.673915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:14.674074 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:14.674654 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:14.674659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:14.674901 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 03:01:14.675075 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 03:01:14.675369 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":358,"threads_started":1}
I20260929 03:01:14.675839 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:14.661216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:47363 (local address 127.2.205.190:35991)
0929 03:01:14.661388 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:14.661394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:14.662248 (+ 854us) server_negotiation.cc:407] Connection header received
0929 03:01:14.662389 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:14.662395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:14.662495 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:14.662607 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:14.663557 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.664818 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.666836 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:14.667463 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:14.667599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:14.670153 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:14.670159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:14.670162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:14.670226 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:14.672705 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.673666 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.673670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.673672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.673799 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:14.674182 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:14.674213 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:14.674215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:14.674532 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:14.674664 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:14.675211 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 03:01:14.675482 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:01:14.685276 4831 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:42723
I20260929 03:01:14.685339 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:42723 every 8 connection(s)
I20260929 03:01:14.687448 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:14.689821 4966 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35991
I20260929 03:01:14.690048 4966 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:14.690683 4966 heartbeater.cc:511] Master 127.2.205.190:35991 requested a full tablet report, sending...
I20260929 03:01:14.692130 4620 ts_manager.cc:194] Registered new tserver with Master: 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:14.693027 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:39279
I20260929 03:01:14.694164 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4831
I20260929 03:01:14.694309 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:14.694875 2870 external_mini_cluster.cc:1428] Setting key 6566f97bedc58c8af40731a672ee0738
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:14.731878 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35991
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:15.054014 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:15.054291 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:15.054390 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:15.061328 4970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:15.061419 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:15.061496 4970 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:01:15.061544 4970 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:01:15.061609 4970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:15.070865 4970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:15.072003 4970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:15.073240 4970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:15.086046 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:15.086086 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:15.086014 4978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:15.090370 4970 server_base.cc:1034] running on GCE node
I20260929 03:01:15.091065 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:15.092133 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:15.093381 4970 hybrid_clock.cc:648] HybridClock initialized: now 1790650875093311 us; error 66 us; skew 500 ppm
I20260929 03:01:15.093722 4970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:15 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650875, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:15.097010 4970 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:15.098619 4970 webserver.cc:460] Webserver started at http://127.2.205.131:36747/ using document root <none> and password file <none>
I20260929 03:01:15.099376 4970 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:15.099509 4970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:15.099808 4970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:15.102913 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "5aa9f7729eb04041a774b252ede32f39"
format_stamp: "Formatted at 2026-09-29 03:01:15 on dist-test-slave-530n"
server_key: "cadb47050fe443a431d61452c2a36fe8"
server_key_iv: "6d936df4f5b818d806b9d77693dcceb7"
server_key_version: "encryptionkey@0"
I20260929 03:01:15.103739 4970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa9f7729eb04041a774b252ede32f39"
format_stamp: "Formatted at 2026-09-29 03:01:15 on dist-test-slave-530n"
server_key: "cadb47050fe443a431d61452c2a36fe8"
server_key_iv: "6d936df4f5b818d806b9d77693dcceb7"
server_key_version: "encryptionkey@0"
I20260929 03:01:15.109614 4970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:01:15.113494 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:15.115051 4970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:15.115206 4970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "5aa9f7729eb04041a774b252ede32f39"
format_stamp: "Formatted at 2026-09-29 03:01:15 on dist-test-slave-530n"
server_key: "cadb47050fe443a431d61452c2a36fe8"
server_key_iv: "6d936df4f5b818d806b9d77693dcceb7"
server_key_version: "encryptionkey@0"
I20260929 03:01:15.115407 4970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:15.138020 4970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:15.142550 4970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:15.142798 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:15.145484 4970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:15.145602 4970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:15.145714 4970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:15.145792 4970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:01:15 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650875, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:15.168431 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.146284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:35027 (local address 127.2.205.190:35991)
0929 03:01:15.146474 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.146480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.148105 (+ 1625us) server_negotiation.cc:407] Connection header received
0929 03:01:15.150190 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.150196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.150289 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.150388 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:15.152901 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.153881 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.156217 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.156821 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.156950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.161416 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:15.161424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:15.161430 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:15.161478 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:15.164810 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.165817 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.165824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.165827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.165991 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.166588 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.166594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.166597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.166913 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:15.167047 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.167843 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.168074 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:01:15.169516 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.147073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.131:35027)
0929 03:01:15.147918 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.147961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.149868 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.150799 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.150811 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.151690 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:15.152669 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.152698 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.154063 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.154067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.156012 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.156024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.157065 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.157077 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.157431 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.158157 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:15.158191 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:15.161100 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:15.164993 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.165003 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.165023 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.165697 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.166104 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.166109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.166113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.166446 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.167048 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:15.167056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:15.167435 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.167708 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.168245 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":377,"threads_started":1}
I20260929 03:01:15.189657 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.175119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.2.205.131:51577)
0929 03:01:15.175770 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.175787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.175926 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.176302 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.176306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.176715 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:15.177121 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.177139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.178571 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.178577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.179698 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.179711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.180818 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.180825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.180996 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.181699 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:15.181718 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:15.182511 (+ 793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:15.186432 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.186437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.186439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.187039 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.187500 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.187505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.187510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.187703 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.188466 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:15.188473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:15.188711 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.188920 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.189317 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":373,"threads_started":1}
I20260929 03:01:15.190428 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.175088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:51577 (local address 127.2.205.190:35991)
0929 03:01:15.175278 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.175283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.175872 (+ 589us) server_negotiation.cc:407] Connection header received
0929 03:01:15.176069 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.176075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.176166 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.176277 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:15.177286 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.178420 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.179854 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.180568 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.180700 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.182762 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:15.182770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:15.182775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:15.182820 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:15.186246 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.187168 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.187174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.187178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.187355 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.187847 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.187853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.187856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.188241 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:15.188395 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.189988 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.190190 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:01:15.199317 4970 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:44527
I20260929 03:01:15.199352 5104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:44527 every 8 connection(s)
I20260929 03:01:15.200585 4970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:15.202278 4827 heartbeater.cc:503] Master 127.2.205.190:35991 was elected leader, sending a full tablet report...
I20260929 03:01:15.202862 5105 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35991
I20260929 03:01:15.203078 5105 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:15.203765 5105 heartbeater.cc:511] Master 127.2.205.190:35991 requested a full tablet report, sending...
I20260929 03:01:15.205138 4620 ts_manager.cc:194] Registered new tserver with Master: 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:15.205981 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:35027
I20260929 03:01:15.208868 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 4970
I20260929 03:01:15.209024 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:15.209421 2870 external_mini_cluster.cc:1428] Setting key e0f16d2f25ce698e1bfc3e78e88945c2
I20260929 03:01:15.212455 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:01:15 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:01:15 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:15.264835 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.249326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.2.205.190:35991)
0929 03:01:15.249525 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.249531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.249795 (+ 264us) server_negotiation.cc:407] Connection header received
0929 03:01:15.249956 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.249960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.250039 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.250127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:15.251948 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.252736 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.254637 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.255285 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.255418 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.260049 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:15.260054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:15.260057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:15.260094 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:15.262469 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.263082 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.263086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.263088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.263221 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.263586 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.263590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.263592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.263900 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:15.264087 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.264326 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.264503 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:01:15.268474 4620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:15.275223 4620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:15.300011 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.287833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.2.205.131:44527)
0929 03:01:15.288543 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.288553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.289164 (+ 611us) server_negotiation.cc:407] Connection header received
0929 03:01:15.289611 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.289620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.289929 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.290136 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:15.291286 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.293200 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.296397 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.298149 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.298273 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.298557 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.299380 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.299603 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":381,"threads_started":1}
I20260929 03:01:15.300949 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.287505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.0.0.1:36320)
0929 03:01:15.288982 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.289037 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.289347 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.290533 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.290542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.290583 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:15.291021 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.291034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.293422 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.293428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.296150 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.296172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.298782 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.298789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.299060 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.299132 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.300259 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.300527 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":1052,"threads_started":1}
I20260929 03:01:15.307037 5035 tablet_service.cc:1467] Processing CreateTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 (DEFAULT_TABLE table=test-table [id=ab5d36a04ae34bdca4f7482f65e56501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:15.308768 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c01044e8d8467e9c3694acc330d5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.308840 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.288124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:42723 (local address 127.0.0.1:58266)
0929 03:01:15.289283 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.289300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.289388 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.302051 (+ 12663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.302056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.302084 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:15.302511 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.302523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.303935 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.303940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.306131 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.306147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.308017 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.308023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.308189 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.308234 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.308333 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.308473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":881,"threads_started":1}
I20260929 03:01:15.308874 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.288688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.2.205.130:42723)
0929 03:01:15.301338 (+ 12650us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.301348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.301385 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:01:15.301502 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.301512 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.301818 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.302062 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:15.302719 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.303803 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.306292 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.307817 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.307956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.308158 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.308407 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.308566 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12436,"thread_start_us":13586,"threads_started":1}
I20260929 03:01:15.310175 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.289522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.0.0.1:33324)
0929 03:01:15.301216 (+ 11694us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.301238 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.301364 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.303662 (+ 2298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.303666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.303692 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:15.303879 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.303889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.305889 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.305894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.307476 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.307489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.309458 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.309464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.309644 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.309675 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.309771 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.309909 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11579,"thread_start_us":13388,"threads_started":1}
I20260929 03:01:15.310227 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.289856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.2.205.129:44633)
0929 03:01:15.302887 (+ 13031us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.302896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.302925 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:01:15.303023 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.303030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.303277 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.303475 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:15.304017 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.305686 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.307641 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.309283 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.309430 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.309626 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.309819 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.309974 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12872,"thread_start_us":12740,"threads_started":1}
I20260929 03:01:15.312885 4896 tablet_service.cc:1467] Processing CreateTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 (DEFAULT_TABLE table=test-table [id=ab5d36a04ae34bdca4f7482f65e56501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:15.313712 4757 tablet_service.cc:1467] Processing CreateTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 (DEFAULT_TABLE table=test-table [id=ab5d36a04ae34bdca4f7482f65e56501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:15.314409 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c01044e8d8467e9c3694acc330d5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.315136 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c01044e8d8467e9c3694acc330d5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.325623 5124 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Bootstrap starting.
I20260929 03:01:15.329866 5126 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Bootstrap starting.
I20260929 03:01:15.330318 5124 tablet_bootstrap.cc:654] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.331881 5124 log.cc:824] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:15.333462 5125 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Bootstrap starting.
I20260929 03:01:15.333633 5126 tablet_bootstrap.cc:654] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.334991 5126 log.cc:824] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:15.336077 5124 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: No bootstrap required, opened a new log
I20260929 03:01:15.336350 5124 ts_tablet_manager.cc:1397] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 03:01:15.337553 5125 tablet_bootstrap.cc:654] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.338971 5125 log.cc:824] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:15.342027 5125 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: No bootstrap required, opened a new log
I20260929 03:01:15.342305 5126 tablet_bootstrap.cc:492] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: No bootstrap required, opened a new log
I20260929 03:01:15.342360 5125 ts_tablet_manager.cc:1397] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 03:01:15.342597 5126 ts_tablet_manager.cc:1397] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260929 03:01:15.345635 5124 raft_consensus.cc:348] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.346100 5124 raft_consensus.cc:374] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.346189 5124 raft_consensus.cc:729] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.346875 5124 consensus_queue.cc:260] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.347959 5105 heartbeater.cc:503] Master 127.2.205.190:35991 was elected leader, sending a full tablet report...
I20260929 03:01:15.349637 5124 ts_tablet_manager.cc:1428] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 03:01:15.350750 5125 raft_consensus.cc:348] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.351051 5125 raft_consensus.cc:374] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.351145 5125 raft_consensus.cc:729] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.351691 5125 consensus_queue.cc:260] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.352110 5126 raft_consensus.cc:348] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.352416 5126 raft_consensus.cc:374] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.352506 5126 raft_consensus.cc:729] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.353111 5126 consensus_queue.cc:260] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.353701 5125 ts_tablet_manager.cc:1428] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 03:01:15.354128 4966 heartbeater.cc:503] Master 127.2.205.190:35991 was elected leader, sending a full tablet report...
I20260929 03:01:15.354653 5126 ts_tablet_manager.cc:1428] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260929 03:01:15.439548 4967 tablet.cc:2369] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:15.445768 5130 raft_consensus.cc:482] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:15.446133 5130 raft_consensus.cc:504] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
W20260929 03:01:15.447683 4828 tablet.cc:2369] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:15.448021 5130 leader_election.cc:288] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
W20260929 03:01:15.452073 5106 tablet.cc:2369] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:15.453989 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.448579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48171 (local address 127.2.205.129:44633)
0929 03:01:15.448812 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.448818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.448915 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:01:15.449174 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.449179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.449253 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.449329 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:15.449722 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.450427 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.452021 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.453129 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.453263 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.453318 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.453684 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.453797 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:01:15.454026 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.448463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.2.205.131:48171)
0929 03:01:15.448817 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.448836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.448981 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.449374 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.449379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.449407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:15.449585 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.449593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.450550 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.450555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.451859 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.451872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.453264 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.453270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.453493 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.453531 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.453629 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.453759 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 03:01:15.454829 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.448877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:55053 (local address 127.2.205.130:42723)
0929 03:01:15.449100 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.449106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.449516 (+ 410us) server_negotiation.cc:407] Connection header received
0929 03:01:15.449689 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.449693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.449760 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.449832 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:15.450286 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.450965 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.452900 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.453911 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.454069 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.454134 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.454514 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.454637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:01:15.454885 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.448469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:42723 (local address 127.2.205.131:55053)
0929 03:01:15.449410 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.449425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.449567 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.449862 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.449866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.449887 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:15.450144 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.450152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.451092 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.451096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.452730 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.452778 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.454146 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.454153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.454356 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.454389 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.454468 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.454554 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":222,"thread_start_us":361,"threads_started":1}
I20260929 03:01:15.455165 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c01044e8d8467e9c3694acc330d5e7" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4" is_pre_election: true
I20260929 03:01:15.455631 4777 raft_consensus.cc:2454] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 0.
I20260929 03:01:15.455760 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c01044e8d8467e9c3694acc330d5e7" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73" is_pre_election: true
I20260929 03:01:15.456168 4916 raft_consensus.cc:2454] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 0.
I20260929 03:01:15.456300 4988 leader_election.cc:302] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:15.456681 5130 raft_consensus.cc:2781] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:15.456838 5130 raft_consensus.cc:482] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:15.456975 5130 raft_consensus.cc:3037] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.459331 5130 raft_consensus.cc:504] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.459869 5130 leader_election.cc:288] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Requested vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:15.460268 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c01044e8d8467e9c3694acc330d5e7" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4"
I20260929 03:01:15.460379 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c01044e8d8467e9c3694acc330d5e7" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73"
I20260929 03:01:15.460487 4777 raft_consensus.cc:3037] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.460562 4916 raft_consensus.cc:3037] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.463595 4916 raft_consensus.cc:2454] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 1.
I20260929 03:01:15.463595 4777 raft_consensus.cc:2454] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 1.
I20260929 03:01:15.464015 4988 leader_election.cc:302] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:15.464308 5130 raft_consensus.cc:2781] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:15.464689 5130 raft_consensus.cc:686] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Running, Role: LEADER
I20260929 03:01:15.465152 5130 consensus_queue.cc:237] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } }
I20260929 03:01:15.469488 4619 catalog_manager.cc:5495] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131). New cstate: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:01:15 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650873, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:15.531704 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.519278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35991 (local address 127.0.0.1:40968)
0929 03:01:15.519899 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.519915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.520058 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.520425 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.520430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.521160 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:15.521437 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.521446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.522495 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.522499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.523447 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.523458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.524174 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.524182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.524321 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.524951 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:15.524972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:15.527020 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:15.529605 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.529615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.529621 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.529970 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.530260 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:15.530264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:15.530266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:15.530424 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:15.530842 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:15.530848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:15.530997 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.531121 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.531336 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":345,"threads_started":1}
I20260929 03:01:15.531798 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.519233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.2.205.190:35991)
0929 03:01:15.519437 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.519444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.519998 (+ 554us) server_negotiation.cc:407] Connection header received
0929 03:01:15.520159 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.520163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.520223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.520325 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:15.521604 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.522376 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.523571 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.524011 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.524102 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.527172 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:15.527176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:15.527179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:15.527207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:15.529510 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.530053 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.530056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.530058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.530172 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:15.530506 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:15.530509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:15.530511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:15.530746 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:15.530855 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.531207 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.531499 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":75}
I20260929 03:01:15.536602 4619 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:40968:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:01:15.537704 4619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:15.544153 4757 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6be256fd214b27b66ab786f15e56bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c26f09f0e61484da9511572c9e85def]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:15.544486 5035 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6be256fd214b27b66ab786f15e56bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c26f09f0e61484da9511572c9e85def]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:15.544320 4896 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6be256fd214b27b66ab786f15e56bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c26f09f0e61484da9511572c9e85def]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:01:15.545104 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6be256fd214b27b66ab786f15e56bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.545264 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6be256fd214b27b66ab786f15e56bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.545292 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6be256fd214b27b66ab786f15e56bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:15.550532 5126 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: Bootstrap starting.
I20260929 03:01:15.551798 5126 tablet_bootstrap.cc:654] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.552276 5125 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: Bootstrap starting.
I20260929 03:01:15.552973 5124 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: Bootstrap starting.
I20260929 03:01:15.553871 5126 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: No bootstrap required, opened a new log
I20260929 03:01:15.553876 5125 tablet_bootstrap.cc:654] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.554000 5126 ts_tablet_manager.cc:1397] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:01:15.554484 5124 tablet_bootstrap.cc:654] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:15.554636 5126 raft_consensus.cc:348] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.554867 5126 raft_consensus.cc:374] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.554958 5126 raft_consensus.cc:729] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.555176 5126 consensus_queue.cc:260] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.555565 5132 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.555940 5132 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.556363 5126 ts_tablet_manager.cc:1428] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:01:15.558306 5124 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: No bootstrap required, opened a new log
I20260929 03:01:15.558445 5124 ts_tablet_manager.cc:1397] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 03:01:15.558528 5125 tablet_bootstrap.cc:492] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: No bootstrap required, opened a new log
I20260929 03:01:15.558662 5125 ts_tablet_manager.cc:1397] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260929 03:01:15.559216 5124 raft_consensus.cc:348] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.559468 5124 raft_consensus.cc:374] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.559365 5125 raft_consensus.cc:348] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.559578 5125 raft_consensus.cc:374] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:15.559577 5124 raft_consensus.cc:729] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.559661 5125 raft_consensus.cc:729] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Initialized, Role: FOLLOWER
I20260929 03:01:15.559808 5124 consensus_queue.cc:260] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.560384 5124 ts_tablet_manager.cc:1428] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 03:01:15.560214 5130 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.560580 5130 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.559930 5125 consensus_queue.cc:260] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.561273 5125 ts_tablet_manager.cc:1428] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 03:01:15.561194 5131 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.561561 5131 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.799636 5130 raft_consensus.cc:482] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:15.799849 5130 raft_consensus.cc:504] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.800490 5130 leader_election.cc:288] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:15.801121 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6be256fd214b27b66ab786f15e56bd" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73" is_pre_election: true
I20260929 03:01:15.801139 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6be256fd214b27b66ab786f15e56bd" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4" is_pre_election: true
I20260929 03:01:15.801394 4916 raft_consensus.cc:2454] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 0.
I20260929 03:01:15.801409 4777 raft_consensus.cc:2454] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 0.
I20260929 03:01:15.801793 4986 leader_election.cc:302] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73; no voters:
I20260929 03:01:15.802084 5130 raft_consensus.cc:2781] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:15.802181 5130 raft_consensus.cc:482] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:15.802285 5130 raft_consensus.cc:3037] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.804205 5130 raft_consensus.cc:504] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.804816 5130 leader_election.cc:288] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Requested vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:15.805229 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6be256fd214b27b66ab786f15e56bd" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4"
I20260929 03:01:15.805285 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6be256fd214b27b66ab786f15e56bd" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73"
I20260929 03:01:15.805441 4777 raft_consensus.cc:3037] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.805513 4916 raft_consensus.cc:3037] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:15.807466 4916 raft_consensus.cc:2454] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 1.
I20260929 03:01:15.807474 4777 raft_consensus.cc:2454] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 1.
I20260929 03:01:15.807845 4986 leader_election.cc:302] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73; no voters:
I20260929 03:01:15.808118 5130 raft_consensus.cc:2781] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:15.808274 5130 raft_consensus.cc:686] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Running, Role: LEADER
I20260929 03:01:15.808511 5130 consensus_queue.cc:237] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:15.809993 5134 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: TxnStatusTablet state changed. Reason: New leader 5aa9f7729eb04041a774b252ede32f39. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.810268 5134 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:15.810822 5146 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:01:15.811103 5146 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:01:15.811506 4620 catalog_manager.cc:5495] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131). New cstate: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260929 03:01:15.862694 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.856432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.0.0.1:36324)
0929 03:01:15.856671 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.856694 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.856857 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.857479 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.857483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.857493 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:15.857637 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.857644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.859102 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.859106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.860096 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.860107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.861062 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.861066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.861225 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.861276 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:15.862299 (+ 1023us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:15.862311 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.862360 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.862466 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:01:15.862869 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.856502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.2.205.131:44527)
0929 03:01:15.857074 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.857084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.857109 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:01:15.857201 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.857206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.857302 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.857470 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:15.857803 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.858980 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.860228 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.860939 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.861059 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.861477 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:15.862207 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:15.862351 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.862396 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.862472 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":269,"threads_started":1}
I20260929 03:01:15.869976 4777 raft_consensus.cc:1261] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Refusing update from remote peer 5aa9f7729eb04041a774b252ede32f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:15.869975 4916 raft_consensus.cc:1261] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Refusing update from remote peer 5aa9f7729eb04041a774b252ede32f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:15.870982 5134 consensus_queue.cc:1035] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:15.871554 5130 consensus_queue.cc:1035] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:15.871641 5151 consensus_queue.cc:1035] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:15.890206 5132 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: TxnStatusTablet state changed. Reason: New leader 5aa9f7729eb04041a774b252ede32f39. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.890561 5132 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.892675 5134 consensus_queue.cc:1035] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:15.900028 5150 mvcc.cc:205] Tried to move back new op lower bound from 7334505987554156544 to 7334505987313885184. Current Snapshot: MvccSnapshot[applied={T|T < 7334505987554156544}]
I20260929 03:01:15.916699 5132 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.917039 5132 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P cce0b0cfec7642b99873fcdd433553c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.917116 5151 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.917486 5151 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:15.917318 5154 mvcc.cc:205] Tried to move back new op lower bound from 7334505987554156544 to 7334505987313885184. Current Snapshot: MvccSnapshot[applied={T|T < 7334505987554156544 or (T in {7334505987554156544})}]
I20260929 03:01:15.928683 5131 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: TxnStatusTablet state changed. Reason: New leader 5aa9f7729eb04041a774b252ede32f39. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.929028 5131 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.929445 5151 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.929703 5151 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 5aa9f7729eb04041a774b252ede32f39: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:01:15.933270 5156 mvcc.cc:205] Tried to move back new op lower bound from 7334505987554156544 to 7334505987313885184. Current Snapshot: MvccSnapshot[applied={T|T < 7334505987554156544}]
I20260929 03:01:15.934131 5131 tablet_replica.cc:439] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aa9f7729eb04041a774b252ede32f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } } }
I20260929 03:01:15.934363 5131 tablet_replica.cc:442] T 6d6be256fd214b27b66ab786f15e56bd P 9364ca482ac640dba903f06dbe54ee73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:01:15.942373 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.935687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.2.205.131:44527)
0929 03:01:15.935855 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.935861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.936522 (+ 661us) server_negotiation.cc:407] Connection header received
0929 03:01:15.936823 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.936829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.936919 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.937018 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:15.937465 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.938613 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.940156 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.940868 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.941024 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.941532 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:15.941676 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:15.941772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.941893 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.941976 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:15.959451 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.954263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48089 (local address 127.2.205.131:44527)
0929 03:01:15.954443 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:15.954449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:15.954891 (+ 442us) server_negotiation.cc:407] Connection header received
0929 03:01:15.955057 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:15.955062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:15.955174 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:15.955281 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:15.955673 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.956422 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.957799 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.958255 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.958354 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.958716 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:15.958815 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:15.958884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:15.958997 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:01:15.959061 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:15.959467 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:15.954401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.2.205.131:48089)
0929 03:01:15.954797 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:15.954811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:15.954946 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:15.955422 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:15.955425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:15.955436 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:15.955553 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.955559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.956543 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.956548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.957675 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:15.957688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:15.958388 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:15.958393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:15.958497 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:15.958541 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:15.958899 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:15.958909 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 03:01:15.958975 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:01:15.959100 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":201,"threads_started":1}
I20260929 03:01:16.027069 4620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:16.027321 4620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:16.031051 4620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Sending DeleteTablet for 3 replicas of tablet 26c01044e8d8467e9c3694acc330d5e7
I20260929 03:01:16.032441 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:16 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58266
I20260929 03:01:16.032787 5034 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:16 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260929 03:01:16.033008 4757 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c01044e8d8467e9c3694acc330d5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:16 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:33324
I20260929 03:01:16.035187 4620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:16.036358 4620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:16.045605 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 3713abb8091f4475b188339e7fede92c (DEFAULT_TABLE table=test-table [id=045ad7c7dcf6443fbda124e1584d998d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:16.045861 5035 tablet_service.cc:1467] Processing CreateTablet for tablet 3713abb8091f4475b188339e7fede92c (DEFAULT_TABLE table=test-table [id=045ad7c7dcf6443fbda124e1584d998d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:16.046236 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3713abb8091f4475b188339e7fede92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:16.046238 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3713abb8091f4475b188339e7fede92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:16.047794 5171 tablet_replica.cc:330] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: stopping tablet replica
I20260929 03:01:16.048486 5171 raft_consensus.cc:2229] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:16.048962 5171 raft_consensus.cc:2258] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:16.049763 5170 tablet_replica.cc:330] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: stopping tablet replica
I20260929 03:01:16.050238 5170 raft_consensus.cc:2229] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:16.050367 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 3713abb8091f4475b188339e7fede92c (DEFAULT_TABLE table=test-table [id=045ad7c7dcf6443fbda124e1584d998d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:16.050863 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3713abb8091f4475b188339e7fede92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:16.051064 5170 raft_consensus.cc:2258] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:16.052476 5126 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Bootstrap starting.
I20260929 03:01:16.053015 5170 ts_tablet_manager.cc:1905] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:16.054163 5126 tablet_bootstrap.cc:654] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:16.054231 5171 ts_tablet_manager.cc:1905] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:16.056963 5124 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Bootstrap starting.
I20260929 03:01:16.058637 5124 tablet_bootstrap.cc:654] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:16.058992 5125 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Bootstrap starting.
I20260929 03:01:16.059376 5169 tablet_replica.cc:330] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: stopping tablet replica
I20260929 03:01:16.059721 5169 raft_consensus.cc:2229] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:16.060082 5169 raft_consensus.cc:2258] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:16.060741 5125 tablet_bootstrap.cc:654] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:16.061676 5126 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: No bootstrap required, opened a new log
I20260929 03:01:16.061893 5126 ts_tablet_manager.cc:1397] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260929 03:01:16.062125 5169 ts_tablet_manager.cc:1905] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:16.062799 5126 raft_consensus.cc:348] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.063125 5126 raft_consensus.cc:374] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:16.063267 5126 raft_consensus.cc:729] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Initialized, Role: FOLLOWER
I20260929 03:01:16.063864 5171 ts_tablet_manager.cc:1918] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:16.064002 5171 log.cc:1196] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/26c01044e8d8467e9c3694acc330d5e7
I20260929 03:01:16.064489 5171 ts_tablet_manager.cc:1939] T 26c01044e8d8467e9c3694acc330d5e7 P cce0b0cfec7642b99873fcdd433553c4: Deleting consensus metadata
I20260929 03:01:16.063620 5126 consensus_queue.cc:260] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.065405 5126 ts_tablet_manager.cc:1428] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:16.066865 4606 catalog_manager.cc:4841] TS cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633): tablet 26c01044e8d8467e9c3694acc330d5e7 (table test-table [id=ab5d36a04ae34bdca4f7482f65e56501]) successfully deleted
I20260929 03:01:16.068850 5169 ts_tablet_manager.cc:1918] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:16.069298 5169 log.cc:1196] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/26c01044e8d8467e9c3694acc330d5e7
I20260929 03:01:16.069814 5169 ts_tablet_manager.cc:1939] T 26c01044e8d8467e9c3694acc330d5e7 P 9364ca482ac640dba903f06dbe54ee73: Deleting consensus metadata
I20260929 03:01:16.070932 5125 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: No bootstrap required, opened a new log
I20260929 03:01:16.071090 5125 ts_tablet_manager.cc:1397] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260929 03:01:16.071554 4604 catalog_manager.cc:4841] TS 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723): tablet 26c01044e8d8467e9c3694acc330d5e7 (table test-table [id=ab5d36a04ae34bdca4f7482f65e56501]) successfully deleted
I20260929 03:01:16.071807 5125 raft_consensus.cc:348] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.072027 5125 raft_consensus.cc:374] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:16.072099 5125 raft_consensus.cc:729] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Initialized, Role: FOLLOWER
I20260929 03:01:16.071970 5170 ts_tablet_manager.cc:1918] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:16.072419 5170 log.cc:1196] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/26c01044e8d8467e9c3694acc330d5e7
I20260929 03:01:16.072350 5125 consensus_queue.cc:260] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.072943 5124 tablet_bootstrap.cc:492] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: No bootstrap required, opened a new log
I20260929 03:01:16.073083 5170 ts_tablet_manager.cc:1939] T 26c01044e8d8467e9c3694acc330d5e7 P 5aa9f7729eb04041a774b252ede32f39: Deleting consensus metadata
I20260929 03:01:16.073107 5124 ts_tablet_manager.cc:1397] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20260929 03:01:16.073876 5124 raft_consensus.cc:348] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.074117 5124 raft_consensus.cc:374] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:16.074205 5124 raft_consensus.cc:729] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Initialized, Role: FOLLOWER
I20260929 03:01:16.074469 5124 consensus_queue.cc:260] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.074890 4606 catalog_manager.cc:4841] TS 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527): tablet 26c01044e8d8467e9c3694acc330d5e7 (table test-table [id=ab5d36a04ae34bdca4f7482f65e56501]) successfully deleted
I20260929 03:01:16.075022 5125 ts_tablet_manager.cc:1428] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 03:01:16.075778 5124 ts_tablet_manager.cc:1428] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:16.186298 5132 raft_consensus.cc:482] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:16.186586 5132 raft_consensus.cc:504] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.188225 5132 leader_election.cc:288] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:16.198125 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.188709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:37011 (local address 127.2.205.131:44527)
0929 03:01:16.188983 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:16.188991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:16.189812 (+ 821us) server_negotiation.cc:407] Connection header received
0929 03:01:16.190124 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:16.190131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:16.190228 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:16.190309 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:16.191186 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.192254 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.195209 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.196654 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.196841 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.196935 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:16.197621 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 03:01:16.197797 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 03:01:16.199030 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.188732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.2.205.129:37011)
0929 03:01:16.189690 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:16.189712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:16.189946 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:16.190525 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:16.190530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:16.190558 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:16.190931 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.190943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.192483 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.192489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.194966 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.194986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.197110 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.197118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.197400 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.197440 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:01:16.198448 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0929 03:01:16.198639 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":606,"threads_started":1}
I20260929 03:01:16.202855 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3713abb8091f4475b188339e7fede92c" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39" is_pre_election: true
I20260929 03:01:16.203388 5055 raft_consensus.cc:2454] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 0.
I20260929 03:01:16.204208 4709 leader_election.cc:302] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:16.204790 5132 raft_consensus.cc:2781] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:16.204922 5132 raft_consensus.cc:482] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:16.205018 5132 raft_consensus.cc:3037] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:16.207947 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.189023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:42723 (local address 127.2.205.129:38493)
0929 03:01:16.199324 (+ 10301us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:16.199348 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:16.199493 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:16.200795 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:16.200801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:16.200837 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:16.201024 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.201036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.202838 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.202845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.204709 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.204731 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.207053 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.207061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.207291 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.207333 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:01:16.207412 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:01:16.207584 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10177,"thread_start_us":14915,"threads_started":1}
I20260929 03:01:16.208370 5132 raft_consensus.cc:504] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.209015 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.188709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:38493 (local address 127.2.205.130:42723)
0929 03:01:16.200405 (+ 11696us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:16.200411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:16.200431 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:16.200502 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:16.200507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:16.200606 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:16.201197 (+ 591us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:16.201453 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.202648 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.205427 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.206801 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.208396 (+ 1595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.208479 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:16.208540 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:01:16.208680 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11566,"thread_start_us":11397,"threads_started":1}
I20260929 03:01:16.209662 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3713abb8091f4475b188339e7fede92c" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73" is_pre_election: true
I20260929 03:01:16.209683 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3713abb8091f4475b188339e7fede92c" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39"
I20260929 03:01:16.209919 5055 raft_consensus.cc:3037] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:16.209932 4915 raft_consensus.cc:2454] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 0.
I20260929 03:01:16.210738 5132 leader_election.cc:288] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:16.211851 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3713abb8091f4475b188339e7fede92c" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73"
I20260929 03:01:16.212066 4915 raft_consensus.cc:3037] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:16.213472 5055 raft_consensus.cc:2454] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 1.
I20260929 03:01:16.213963 4709 leader_election.cc:302] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:16.214346 5132 raft_consensus.cc:2781] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:16.215116 4915 raft_consensus.cc:2454] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 1.
I20260929 03:01:16.225457 5132 raft_consensus.cc:686] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 1 LEADER]: Becoming Leader. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Running, Role: LEADER
I20260929 03:01:16.225980 5132 consensus_queue.cc:237] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:16.229811 4618 catalog_manager.cc:5495] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "cce0b0cfec7642b99873fcdd433553c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:16.272637 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.266306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.2.205.129:44633)
0929 03:01:16.266834 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:16.266838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:16.266856 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:16.266929 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:16.266933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:16.267006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:16.267135 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:16.267507 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.268280 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.270259 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.270708 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.270808 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.271329 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:16.271940 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:16.272047 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:16.272219 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:01:16.272295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":280,"threads_started":1}
I20260929 03:01:16.288842 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.282511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.2.205.129:49087)
0929 03:01:16.283193 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:16.283210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:16.283338 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:16.283668 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:16.283671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:16.283680 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:16.283983 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.283992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.285557 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.285561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.286638 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.286652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.287640 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.287644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.287776 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.287845 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:16.288289 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:16.288303 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:01:16.288379 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:01:16.288494 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":348,"threads_started":1}
I20260929 03:01:16.288830 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.282525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:49087 (local address 127.2.205.131:44527)
0929 03:01:16.282701 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:16.282706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:16.283295 (+ 589us) server_negotiation.cc:407] Connection header received
0929 03:01:16.283469 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:16.283475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:16.283566 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:16.283664 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:16.284105 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.285354 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.286782 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.287473 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.287596 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.288009 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:16.288164 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:16.288251 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:16.288417 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 03:01:16.288506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:01:16.298960 4915 raft_consensus.cc:1261] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Refusing update from remote peer cce0b0cfec7642b99873fcdd433553c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:16.299717 5055 raft_consensus.cc:1261] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Refusing update from remote peer cce0b0cfec7642b99873fcdd433553c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:16.299714 5178 consensus_queue.cc:1035] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:16.300580 5178 consensus_queue.cc:1035] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:16.336339 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.330369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.2.205.131:52937)
0929 03:01:16.330564 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:16.330585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:16.330688 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:16.331434 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:16.331439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:16.331453 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:16.331652 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.331671 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.333079 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.333085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.334399 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:16.334412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.335220 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.335226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:16.335404 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.335459 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:16.335828 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:16.335841 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:01:16.335919 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:01:16.336066 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 03:01:16.336351 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:16.330844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:52937 (local address 127.2.205.129:44633)
0929 03:01:16.331004 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:16.331009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:16.331027 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:16.331083 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:16.331087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:16.331167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:16.331250 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:16.331808 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.332925 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.334527 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:16.335074 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:16.335193 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:16.335617 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:16.335736 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:16.335811 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:16.335971 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:01:16.336062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 03:01:16.337678 5154 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:01:17.940626 4620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:17.940932 4620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:17.945266 4620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Sending DeleteTablet for 3 replicas of tablet 3713abb8091f4475b188339e7fede92c
I20260929 03:01:17.946179 4756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3713abb8091f4475b188339e7fede92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:33324
I20260929 03:01:17.946208 5035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3713abb8091f4475b188339e7fede92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260929 03:01:17.946185 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3713abb8091f4475b188339e7fede92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58266
I20260929 03:01:17.956416 5213 tablet_replica.cc:330] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: stopping tablet replica
I20260929 03:01:17.956725 5213 raft_consensus.cc:2229] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:17.957032 5213 raft_consensus.cc:2258] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:17.957787 5211 tablet_replica.cc:330] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: stopping tablet replica
I20260929 03:01:17.958026 5211 raft_consensus.cc:2229] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:17.958581 5213 ts_tablet_manager.cc:1905] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:17.958581 5211 raft_consensus.cc:2258] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:17.958904 5212 tablet_replica.cc:330] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: stopping tablet replica
I20260929 03:01:17.959079 5212 raft_consensus.cc:2229] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:17.959296 5212 raft_consensus.cc:2258] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:17.959619 5211 ts_tablet_manager.cc:1905] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:17.960132 5212 ts_tablet_manager.cc:1905] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:17.965878 5213 ts_tablet_manager.cc:1918] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:17.966022 5213 log.cc:1196] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/3713abb8091f4475b188339e7fede92c
I20260929 03:01:17.966271 5212 ts_tablet_manager.cc:1918] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:17.966416 5212 log.cc:1196] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/3713abb8091f4475b188339e7fede92c
I20260929 03:01:17.966463 5213 ts_tablet_manager.cc:1939] T 3713abb8091f4475b188339e7fede92c P 9364ca482ac640dba903f06dbe54ee73: Deleting consensus metadata
I20260929 03:01:17.966674 5211 ts_tablet_manager.cc:1918] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:17.966811 5212 ts_tablet_manager.cc:1939] T 3713abb8091f4475b188339e7fede92c P 5aa9f7729eb04041a774b252ede32f39: Deleting consensus metadata
I20260929 03:01:17.966857 5211 log.cc:1196] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/3713abb8091f4475b188339e7fede92c
I20260929 03:01:17.967397 5211 ts_tablet_manager.cc:1939] T 3713abb8091f4475b188339e7fede92c P cce0b0cfec7642b99873fcdd433553c4: Deleting consensus metadata
I20260929 03:01:17.967806 4606 catalog_manager.cc:4841] TS 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527): tablet 3713abb8091f4475b188339e7fede92c (table test-table [id=045ad7c7dcf6443fbda124e1584d998d]) successfully deleted
I20260929 03:01:17.967759 4604 catalog_manager.cc:4841] TS 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723): tablet 3713abb8091f4475b188339e7fede92c (table test-table [id=045ad7c7dcf6443fbda124e1584d998d]) successfully deleted
I20260929 03:01:17.968850 4606 catalog_manager.cc:4841] TS cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633): tablet 3713abb8091f4475b188339e7fede92c (table test-table [id=045ad7c7dcf6443fbda124e1584d998d]) successfully deleted
Sep 29 03:01:17 dist-test-slave-530n krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:01:17 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:18.006210 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:17.985159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.2.205.190:35991)
0929 03:01:17.985756 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:17.985761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:17.985786 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:01:17.985975 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:17.985980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:17.986070 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:17.986204 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:17.987605 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:17.988404 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:17.989754 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:17.990185 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:17.990289 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.000709 (+ 10420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:18.000715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:18.000719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:18.000796 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:18.003451 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:18.004172 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:18.004178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:18.004180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:18.004316 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:18.004705 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:18.004710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:18.004713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:18.005077 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:18.005328 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.005624 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.005854 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":307,"threads_started":1}
I20260929 03:01:18.009671 4620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:18.010293 4620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:18.016876 4756 tablet_service.cc:1467] Processing CreateTablet for tablet f096f0793df745ad9de64a7b8482989b (DEFAULT_TABLE table=test-table [id=a244999d3f0445cba039306f443290b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.017010 4896 tablet_service.cc:1467] Processing CreateTablet for tablet f096f0793df745ad9de64a7b8482989b (DEFAULT_TABLE table=test-table [id=a244999d3f0445cba039306f443290b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.017095 5035 tablet_service.cc:1467] Processing CreateTablet for tablet f096f0793df745ad9de64a7b8482989b (DEFAULT_TABLE table=test-table [id=a244999d3f0445cba039306f443290b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.017465 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f096f0793df745ad9de64a7b8482989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.017504 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f096f0793df745ad9de64a7b8482989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.017496 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f096f0793df745ad9de64a7b8482989b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.024219 5223 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Bootstrap starting.
I20260929 03:01:18.025054 5224 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Bootstrap starting.
I20260929 03:01:18.025359 5222 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Bootstrap starting.
I20260929 03:01:18.026010 5223 tablet_bootstrap.cc:654] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.026676 5224 tablet_bootstrap.cc:654] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.027014 5222 tablet_bootstrap.cc:654] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.028714 5223 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: No bootstrap required, opened a new log
I20260929 03:01:18.028872 5223 ts_tablet_manager.cc:1397] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 03:01:18.029130 5224 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: No bootstrap required, opened a new log
I20260929 03:01:18.029247 5224 ts_tablet_manager.cc:1397] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 03:01:18.029518 5223 raft_consensus.cc:348] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.029716 5223 raft_consensus.cc:374] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.029803 5223 raft_consensus.cc:729] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.029824 5224 raft_consensus.cc:348] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.030028 5224 raft_consensus.cc:374] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.030113 5224 raft_consensus.cc:729] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.030030 5223 consensus_queue.cc:260] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.030326 5224 consensus_queue.cc:260] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.030596 5223 ts_tablet_manager.cc:1428] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:01:18.030925 5224 ts_tablet_manager.cc:1428] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:01:18.031939 5222 tablet_bootstrap.cc:492] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: No bootstrap required, opened a new log
I20260929 03:01:18.032063 5222 ts_tablet_manager.cc:1397] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 03:01:18.032734 5222 raft_consensus.cc:348] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.032982 5222 raft_consensus.cc:374] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.033071 5222 raft_consensus.cc:729] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.033291 5222 consensus_queue.cc:260] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.034516 5222 ts_tablet_manager.cc:1428] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:01:18.082783 5201 raft_consensus.cc:482] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:18.083019 5201 raft_consensus.cc:504] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.083766 5201 leader_election.cc:288] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:18.084367 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f096f0793df745ad9de64a7b8482989b" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73" is_pre_election: true
I20260929 03:01:18.084407 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f096f0793df745ad9de64a7b8482989b" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39" is_pre_election: true
I20260929 03:01:18.084620 4915 raft_consensus.cc:2454] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 0.
I20260929 03:01:18.084661 5055 raft_consensus.cc:2454] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 0.
I20260929 03:01:18.085182 4707 leader_election.cc:302] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:18.085505 5201 raft_consensus.cc:2781] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:18.085624 5201 raft_consensus.cc:482] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:18.085696 5201 raft_consensus.cc:3037] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.088295 5201 raft_consensus.cc:504] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.088937 5201 leader_election.cc:288] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:18.089521 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f096f0793df745ad9de64a7b8482989b" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39"
I20260929 03:01:18.089517 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f096f0793df745ad9de64a7b8482989b" candidate_uuid: "cce0b0cfec7642b99873fcdd433553c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73"
I20260929 03:01:18.089751 4915 raft_consensus.cc:3037] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.089742 5055 raft_consensus.cc:3037] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.092648 5055 raft_consensus.cc:2454] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 1.
I20260929 03:01:18.092648 4915 raft_consensus.cc:2454] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce0b0cfec7642b99873fcdd433553c4 in term 1.
I20260929 03:01:18.093212 4709 leader_election.cc:302] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:18.093542 5201 raft_consensus.cc:2781] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:18.093681 5201 raft_consensus.cc:686] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 1 LEADER]: Becoming Leader. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Running, Role: LEADER
I20260929 03:01:18.093986 5201 consensus_queue.cc:237] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.096596 4618 catalog_manager.cc:5495] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "cce0b0cfec7642b99873fcdd433553c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:18.136211 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.131011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.2.205.129:44633)
0929 03:01:18.131611 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.131615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.131633 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:18.131690 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.131693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.131763 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.131843 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:18.132145 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.132961 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.134263 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.134664 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.134760 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.135126 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.135240 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.135303 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.135418 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.135479 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260929 03:01:18.145682 5055 raft_consensus.cc:1261] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Refusing update from remote peer cce0b0cfec7642b99873fcdd433553c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:18.145783 4915 raft_consensus.cc:1261] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Refusing update from remote peer cce0b0cfec7642b99873fcdd433553c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:18.146230 5204 consensus_queue.cc:1035] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:18.146554 5201 consensus_queue.cc:1035] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:18.189196 4618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:18.189416 4618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:18.192414 4618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Sending DeleteTablet for 3 replicas of tablet f096f0793df745ad9de64a7b8482989b
I20260929 03:01:18.193230 4756 tablet_service.cc:1514] Processing DeleteTablet for tablet f096f0793df745ad9de64a7b8482989b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:18 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:33324
I20260929 03:01:18.193315 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet f096f0793df745ad9de64a7b8482989b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:18 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58266
I20260929 03:01:18.193514 5211 tablet_replica.cc:330] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: stopping tablet replica
I20260929 03:01:18.193550 5213 tablet_replica.cc:330] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: stopping tablet replica
I20260929 03:01:18.193728 5213 raft_consensus.cc:2229] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:18.193728 5211 raft_consensus.cc:2229] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:18.193912 5213 raft_consensus.cc:2258] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:18.194056 5211 raft_consensus.cc:2258] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:18.194417 5035 tablet_service.cc:1514] Processing DeleteTablet for tablet f096f0793df745ad9de64a7b8482989b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:18 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260929 03:01:18.194795 5212 tablet_replica.cc:330] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: stopping tablet replica
I20260929 03:01:18.194691 4618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:01:18.194988 5212 raft_consensus.cc:2229] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:18.195161 5212 raft_consensus.cc:2258] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:18.195233 5211 ts_tablet_manager.cc:1905] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:01:18.195355 4618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:18.196290 5213 ts_tablet_manager.cc:1905] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:18.197748 5212 ts_tablet_manager.cc:1905] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:18.202777 5213 ts_tablet_manager.cc:1918] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:18.202916 5213 log.cc:1196] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/f096f0793df745ad9de64a7b8482989b
I20260929 03:01:18.203336 5213 ts_tablet_manager.cc:1939] T f096f0793df745ad9de64a7b8482989b P 9364ca482ac640dba903f06dbe54ee73: Deleting consensus metadata
I20260929 03:01:18.204175 5212 ts_tablet_manager.cc:1918] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:18.204321 5212 log.cc:1196] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/f096f0793df745ad9de64a7b8482989b
I20260929 03:01:18.204677 4604 catalog_manager.cc:4841] TS 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723): tablet f096f0793df745ad9de64a7b8482989b (table test-table [id=a244999d3f0445cba039306f443290b9]) successfully deleted
I20260929 03:01:18.204793 5212 ts_tablet_manager.cc:1939] T f096f0793df745ad9de64a7b8482989b P 5aa9f7729eb04041a774b252ede32f39: Deleting consensus metadata
I20260929 03:01:18.205246 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 660163fb4c26474783c92806143026df (DEFAULT_TABLE table=test-table [id=4b7a1eadd28340be981a2f2a8181f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.205374 4896 tablet_service.cc:1467] Processing CreateTablet for tablet 660163fb4c26474783c92806143026df (DEFAULT_TABLE table=test-table [id=4b7a1eadd28340be981a2f2a8181f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.205655 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660163fb4c26474783c92806143026df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.205740 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660163fb4c26474783c92806143026df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.205706 4606 catalog_manager.cc:4841] TS 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527): tablet f096f0793df745ad9de64a7b8482989b (table test-table [id=a244999d3f0445cba039306f443290b9]) successfully deleted
I20260929 03:01:18.209028 5035 tablet_service.cc:1467] Processing CreateTablet for tablet 660163fb4c26474783c92806143026df (DEFAULT_TABLE table=test-table [id=4b7a1eadd28340be981a2f2a8181f5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:18.209412 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660163fb4c26474783c92806143026df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:18.211918 5222 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Bootstrap starting.
I20260929 03:01:18.212232 5211 ts_tablet_manager.cc:1918] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:01:18.212386 5211 log.cc:1196] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/f096f0793df745ad9de64a7b8482989b
I20260929 03:01:18.212790 5211 ts_tablet_manager.cc:1939] T f096f0793df745ad9de64a7b8482989b P cce0b0cfec7642b99873fcdd433553c4: Deleting consensus metadata
I20260929 03:01:18.213466 5222 tablet_bootstrap.cc:654] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.213726 4606 catalog_manager.cc:4841] TS cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633): tablet f096f0793df745ad9de64a7b8482989b (table test-table [id=a244999d3f0445cba039306f443290b9]) successfully deleted
I20260929 03:01:18.215322 5223 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Bootstrap starting.
I20260929 03:01:18.216822 5223 tablet_bootstrap.cc:654] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.221166 5224 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Bootstrap starting.
I20260929 03:01:18.223176 5224 tablet_bootstrap.cc:654] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:18.225946 5222 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: No bootstrap required, opened a new log
I20260929 03:01:18.226128 5222 ts_tablet_manager.cc:1397] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20260929 03:01:18.226954 5222 raft_consensus.cc:348] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.227240 5222 raft_consensus.cc:374] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.227316 5222 raft_consensus.cc:729] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.227612 5222 consensus_queue.cc:260] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.228209 5222 ts_tablet_manager.cc:1428] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:01:18.229746 5223 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: No bootstrap required, opened a new log
I20260929 03:01:18.229933 5223 ts_tablet_manager.cc:1397] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260929 03:01:18.230751 5223 raft_consensus.cc:348] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.231012 5223 raft_consensus.cc:374] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.231103 5223 raft_consensus.cc:729] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa9f7729eb04041a774b252ede32f39, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.231391 5223 consensus_queue.cc:260] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.232049 5223 ts_tablet_manager.cc:1428] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 03:01:18.236198 5224 tablet_bootstrap.cc:492] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: No bootstrap required, opened a new log
I20260929 03:01:18.236374 5224 ts_tablet_manager.cc:1397] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260929 03:01:18.237046 5224 raft_consensus.cc:348] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.237254 5224 raft_consensus.cc:374] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:18.237349 5224 raft_consensus.cc:729] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce0b0cfec7642b99873fcdd433553c4, State: Initialized, Role: FOLLOWER
I20260929 03:01:18.237602 5224 consensus_queue.cc:260] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.238195 5224 ts_tablet_manager.cc:1428] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:01:18.566852 5228 raft_consensus.cc:482] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:18.567083 5228 raft_consensus.cc:504] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.568642 5228 leader_election.cc:288] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:18.576032 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.569038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:41259 (local address 127.2.205.129:44633)
0929 03:01:18.569204 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.569208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.569730 (+ 522us) server_negotiation.cc:407] Connection header received
0929 03:01:18.569951 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.569958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.570036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.570120 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:18.570778 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.571750 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.574093 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.575048 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.575211 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.575298 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.575643 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.575788 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:01:18.576687 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.569101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.2.205.130:41259)
0929 03:01:18.569611 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:18.569629 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:18.569804 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:18.570286 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:18.570303 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:18.570333 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:18.570617 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.570627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.571918 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.571923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.573932 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.573949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.575286 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.575292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.575468 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.575497 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:01:18.575592 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:01:18.575734 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260929 03:01:18.577054 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.569718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.2.205.130:38185)
0929 03:01:18.570288 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:18.570304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:18.570418 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:18.570773 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:18.570776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:18.570803 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:18.571100 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.571109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.572782 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.572788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.574496 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.574507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.576091 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.576096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.576274 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.576304 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:01:18.576418 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:01:18.576712 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":298,"threads_started":1}
I20260929 03:01:18.577131 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.569877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:38185 (local address 127.2.205.131:44527)
0929 03:01:18.570517 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.570521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.570542 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:01:18.570598 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.570601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.570655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.570753 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:18.571257 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.572597 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.574641 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.575784 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.575966 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.576038 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.576442 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.576582 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":415,"threads_started":1}
I20260929 03:01:18.577554 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "9364ca482ac640dba903f06dbe54ee73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4" is_pre_election: true
I20260929 03:01:18.577648 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "9364ca482ac640dba903f06dbe54ee73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39" is_pre_election: true
I20260929 03:01:18.577800 4776 raft_consensus.cc:2454] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9364ca482ac640dba903f06dbe54ee73 in term 0.
I20260929 03:01:18.577886 5055 raft_consensus.cc:2454] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9364ca482ac640dba903f06dbe54ee73 in term 0.
I20260929 03:01:18.578028 5151 raft_consensus.cc:482] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:18.578217 5151 raft_consensus.cc:504] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.578374 4849 leader_election.cc:302] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, 9364ca482ac640dba903f06dbe54ee73; no voters:
I20260929 03:01:18.578809 5151 leader_election.cc:288] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:18.578984 5228 raft_consensus.cc:2781] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:18.579111 5228 raft_consensus.cc:482] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:18.579208 5228 raft_consensus.cc:3037] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.579298 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73" is_pre_election: true
I20260929 03:01:18.579439 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4" is_pre_election: true
I20260929 03:01:18.579665 4776 raft_consensus.cc:2454] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa9f7729eb04041a774b252ede32f39 in term 0.
I20260929 03:01:18.580089 4988 leader_election.cc:302] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39, cce0b0cfec7642b99873fcdd433553c4; no voters:
I20260929 03:01:18.580390 5151 raft_consensus.cc:2781] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:18.580490 5151 raft_consensus.cc:482] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:18.580596 5151 raft_consensus.cc:3037] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.581727 5228 raft_consensus.cc:504] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.582137 4916 raft_consensus.cc:2379] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa9f7729eb04041a774b252ede32f39 in current term 1: Already voted for candidate 9364ca482ac640dba903f06dbe54ee73 in this term.
I20260929 03:01:18.582748 5151 raft_consensus.cc:504] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.583073 5228 leader_election.cc:288] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [CANDIDATE]: Term 1 election: Requested vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527)
I20260929 03:01:18.583581 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "9364ca482ac640dba903f06dbe54ee73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4"
I20260929 03:01:18.583602 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "9364ca482ac640dba903f06dbe54ee73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa9f7729eb04041a774b252ede32f39"
I20260929 03:01:18.583802 4776 raft_consensus.cc:3037] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:18.583873 5055 raft_consensus.cc:2379] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9364ca482ac640dba903f06dbe54ee73 in current term 1: Already voted for candidate 5aa9f7729eb04041a774b252ede32f39 in this term.
I20260929 03:01:18.584226 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce0b0cfec7642b99873fcdd433553c4"
I20260929 03:01:18.584208 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660163fb4c26474783c92806143026df" candidate_uuid: "5aa9f7729eb04041a774b252ede32f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9364ca482ac640dba903f06dbe54ee73"
I20260929 03:01:18.584486 4916 raft_consensus.cc:2379] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5aa9f7729eb04041a774b252ede32f39 in current term 1: Already voted for candidate 9364ca482ac640dba903f06dbe54ee73 in this term.
I20260929 03:01:18.585040 4986 leader_election.cc:302] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5aa9f7729eb04041a774b252ede32f39; no voters: 9364ca482ac640dba903f06dbe54ee73, cce0b0cfec7642b99873fcdd433553c4
I20260929 03:01:18.585304 5151 leader_election.cc:288] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [CANDIDATE]: Term 1 election: Requested vote from peers cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633), 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723)
I20260929 03:01:18.585361 5233 raft_consensus.cc:2726] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 03:01:18.586606 4776 raft_consensus.cc:2454] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9364ca482ac640dba903f06dbe54ee73 in term 1.
I20260929 03:01:18.586973 4849 leader_election.cc:302] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9364ca482ac640dba903f06dbe54ee73, cce0b0cfec7642b99873fcdd433553c4; no voters: 5aa9f7729eb04041a774b252ede32f39
I20260929 03:01:18.587242 5228 raft_consensus.cc:2781] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:18.587682 5228 raft_consensus.cc:686] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 LEADER]: Becoming Leader. State: Replica: 9364ca482ac640dba903f06dbe54ee73, State: Running, Role: LEADER
I20260929 03:01:18.588207 5228 consensus_queue.cc:237] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } }
I20260929 03:01:18.591495 4620 catalog_manager.cc:5495] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130). New cstate: current_term: 1 leader_uuid: "9364ca482ac640dba903f06dbe54ee73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9364ca482ac640dba903f06dbe54ee73" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:18.681649 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.676074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.2.205.130:42723)
0929 03:01:18.676613 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.676617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.676677 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:01:18.676888 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.676892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.676963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.677096 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:18.677474 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.678234 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.679508 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.679962 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.680056 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.680394 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.681090 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.681220 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.681266 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.681327 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":309,"threads_started":1}
I20260929 03:01:18.698356 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.691147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44527 (local address 127.2.205.130:43615)
0929 03:01:18.691995 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:18.692014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:18.692153 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:18.692457 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:18.692461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:18.692473 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:18.693313 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.693326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.694327 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.694332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.696097 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.696114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.697023 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.697029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.697221 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.697295 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.697659 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.697670 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:01:18.697774 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:01:18.697943 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":487,"threads_started":1}
I20260929 03:01:18.698406 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.691294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:43615 (local address 127.2.205.131:44527)
0929 03:01:18.691481 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.691488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.692098 (+ 610us) server_negotiation.cc:407] Connection header received
0929 03:01:18.692289 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.692293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.692359 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.692429 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:18.693485 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.694193 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.696287 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.696709 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.696892 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.697456 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.697571 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.697638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.697881 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.698002 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:01:18.710150 5055 raft_consensus.cc:1261] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Refusing update from remote peer 9364ca482ac640dba903f06dbe54ee73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:18.710994 4776 raft_consensus.cc:1261] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Refusing update from remote peer 9364ca482ac640dba903f06dbe54ee73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:18.710861 5242 consensus_queue.cc:1035] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa9f7729eb04041a774b252ede32f39" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:18.711825 5242 consensus_queue.cc:1035] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce0b0cfec7642b99873fcdd433553c4" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:18.741082 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.735919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:52017 (local address 127.2.205.130:42723)
0929 03:01:18.736085 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:18.736091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:18.736542 (+ 451us) server_negotiation.cc:407] Connection header received
0929 03:01:18.736735 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:18.736741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:18.736845 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:18.736932 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:18.737245 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.737921 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.739459 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.739900 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.740005 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.740394 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.740494 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.740565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:18.740686 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:01:18.740799 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:01:18.741147 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:18.735899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:42723 (local address 127.2.205.131:52017)
0929 03:01:18.736452 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:18.736467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:18.736596 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:18.736954 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:18.736957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:18.736967 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:01:18.737140 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.737147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.738042 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.738047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.739332 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:18.739343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:18.740063 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:18.740070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:18.740215 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:18.740252 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:01:18.740577 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:01:18.740589 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:01:18.740658 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:01:18.740814 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":296,"threads_started":1}
W20260929 03:01:18.742188 5192 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:01:20.326047 4620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:20.326274 4620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:20.330226 4620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04409471a2264ec780a7cae16bcd8528: Sending DeleteTablet for 3 replicas of tablet 660163fb4c26474783c92806143026df
I20260929 03:01:20.331071 4756 tablet_service.cc:1514] Processing DeleteTablet for tablet 660163fb4c26474783c92806143026df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:20 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:33324
I20260929 03:01:20.331065 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet 660163fb4c26474783c92806143026df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:20 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58266
I20260929 03:01:20.331174 5035 tablet_service.cc:1514] Processing DeleteTablet for tablet 660163fb4c26474783c92806143026df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:20 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260929 03:01:20.331915 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4691
I20260929 03:01:20.343576 5282 tablet_replica.cc:330] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: stopping tablet replica
I20260929 03:01:20.343568 5281 tablet_replica.cc:330] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: stopping tablet replica
I20260929 03:01:20.343806 5281 raft_consensus.cc:2229] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:20.343793 5282 raft_consensus.cc:2229] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:20.344064 5282 raft_consensus.cc:2258] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:20.344396 5281 raft_consensus.cc:2258] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:20.345484 5281 ts_tablet_manager.cc:1905] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:20.345530 5282 ts_tablet_manager.cc:1905] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:20.350696 5280 tablet_replica.cc:330] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: stopping tablet replica
I20260929 03:01:20.350900 5280 raft_consensus.cc:2229] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:20.351159 5280 raft_consensus.cc:2258] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:20.351972 5281 ts_tablet_manager.cc:1918] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:20.352099 5281 log.cc:1196] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/660163fb4c26474783c92806143026df
I20260929 03:01:20.352344 5280 ts_tablet_manager.cc:1905] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:20.352555 5281 ts_tablet_manager.cc:1939] T 660163fb4c26474783c92806143026df P 9364ca482ac640dba903f06dbe54ee73: Deleting consensus metadata
I20260929 03:01:20.353405 5282 ts_tablet_manager.cc:1918] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:20.353536 5282 log.cc:1196] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/660163fb4c26474783c92806143026df
I20260929 03:01:20.353708 4604 catalog_manager.cc:4841] TS 9364ca482ac640dba903f06dbe54ee73 (127.2.205.130:42723): tablet 660163fb4c26474783c92806143026df (table test-table [id=4b7a1eadd28340be981a2f2a8181f5f4]) successfully deleted
I20260929 03:01:20.353958 5282 ts_tablet_manager.cc:1939] T 660163fb4c26474783c92806143026df P 5aa9f7729eb04041a774b252ede32f39: Deleting consensus metadata
I20260929 03:01:20.355077 4606 catalog_manager.cc:4841] TS 5aa9f7729eb04041a774b252ede32f39 (127.2.205.131:44527): tablet 660163fb4c26474783c92806143026df (table test-table [id=4b7a1eadd28340be981a2f2a8181f5f4]) successfully deleted
I20260929 03:01:20.356989 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.333898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.2.205.129:44633)
0929 03:01:20.335358 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:20.335365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:20.335391 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:01:20.335460 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:20.335467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:20.335551 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:20.335665 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:20.337708 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.339061 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.341204 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.342044 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.342199 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:20.349401 (+ 7202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:20.349424 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:20.349445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:20.349494 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:20.353448 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.354321 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.354334 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.354337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.354526 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.354973 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.354978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.354981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.355359 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:20.355659 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:20.355937 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:01:20.356163 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1147,"threads_started":1}
W20260929 03:01:20.357600 4822 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33346
W20260929 03:01:20.358175 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:20.358225 5280 ts_tablet_manager.cc:1918] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:01:20.358327 5280 log.cc:1196] T 660163fb4c26474783c92806143026df P cce0b0cfec7642b99873fcdd433553c4: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/660163fb4c26474783c92806143026df
I20260929 03:01:20.358392 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4691
W20260929 03:01:20.375459 4606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:44633 (error 108)
I20260929 03:01:20.375691 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4831
I20260929 03:01:20.377638 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.376730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.0.0.1:33348)
0929 03:01:20.377248 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:20.377381 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":245,"threads_started":1}
W20260929 03:01:20.377868 4606 catalog_manager.cc:4584] TS cce0b0cfec7642b99873fcdd433553c4 (127.2.205.129:44633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 660163fb4c26474783c92806143026df: Network error: Client connection negotiation failed: client connection to 127.2.205.129:44633: connect: Connection refused (error 111)
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:20.391104 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.376361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.2.205.130:42723)
0929 03:01:20.377156 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:20.377160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:20.377175 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:01:20.377225 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:20.377229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:20.377286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:20.377370 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:20.378738 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.379472 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.380651 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.381231 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.381352 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:20.385416 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:20.385432 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:20.385450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:20.385490 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:20.388352 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.389167 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.389177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.389179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.389360 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.389663 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.389667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.389669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.390085 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:20.390333 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:20.390580 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:01:20.390764 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":574,"threads_started":1}
W20260929 03:01:20.391780 4961 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58284
W20260929 03:01:20.392484 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4831: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:20.392622 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4831
I20260929 03:01:20.410092 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4970
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:20.426203 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.410839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.2.205.131:44527)
0929 03:01:20.411371 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:20.411375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:20.411393 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:20.411448 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:20.411452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:20.411512 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:20.411602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:20.412920 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.413690 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.414959 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:20.415375 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:20.415468 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:20.420235 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:20.420252 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:20.420270 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:20.420321 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:20.423196 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.423995 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.424004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.424007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.424166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:20.424555 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:20.424559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:20.424561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:20.425006 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:20.425308 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:20.425600 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 03:01:20.425832 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":252,"threads_started":1}
W20260929 03:01:20.426879 5100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36342
W20260929 03:01:20.427536 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4970: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:20.427696 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4970
I20260929 03:01:20.431248 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.430814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.0.0.1:33354)
0929 03:01:20.431012 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:20.431098 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 03:01:20.432227 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.431853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.0.0.1:33368)
0929 03:01:20.432001 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:20.432088 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:01:20.449186 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 4588
I20260929 03:01:20.450361 4660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:20.565615 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:20.565156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44633 (local address 127.0.0.1:33384)
0929 03:01:20.565369 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:20.565451 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:01:20.565930 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 4588
2026-09-29T03:01:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8518 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[5292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[5292](info): set up 2 sockets
Sep 29 03:01:20 dist-test-slave-530n krb5kdc[5292](info): commencing operation
krb5kdc: starting...
W20260929 03:01:21.565126 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:21 dist-test-slave-530n krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:21.744576 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46295
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:22.060204 5305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:22.060490 5305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:22.060627 5305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:22.067673 5305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:22.067785 5305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:22.067862 5305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:22.067937 5305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:22.067996 5305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:22.077620 5305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:22.078933 5305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:22.080248 5305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:22.092108 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:22.092144 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:22.092190 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:22.092486 5305 server_base.cc:1034] running on GCE node
I20260929 03:01:22.093561 5305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:22.095091 5305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:22.096312 5305 hybrid_clock.cc:648] HybridClock initialized: now 1790650882096254 us; error 48 us; skew 500 ppm
I20260929 03:01:22.096674 5305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:22 dist-test-slave-530n krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:22.099839 5305 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:22.101471 5305 webserver.cc:460] Webserver started at http://127.2.205.190:43665/ using document root <none> and password file <none>
I20260929 03:01:22.102166 5305 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:22.102326 5305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:22.102626 5305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:22.105723 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "fbb0741b5600416eae01219fa497548e"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "cfbcd0e4b554713f5edc13fbc241dee6"
server_key_iv: "35953c588c75f79c8b9d59d80d1fc217"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.106530 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "fbb0741b5600416eae01219fa497548e"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "cfbcd0e4b554713f5edc13fbc241dee6"
server_key_iv: "35953c588c75f79c8b9d59d80d1fc217"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.112088 5305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:01:22.115972 5320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:22.117628 5305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:22.117827 5305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "fbb0741b5600416eae01219fa497548e"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "cfbcd0e4b554713f5edc13fbc241dee6"
server_key_iv: "35953c588c75f79c8b9d59d80d1fc217"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.118036 5305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:22.139122 5305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:22.143798 5305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:22.144093 5305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:22.161450 5305 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:46295
I20260929 03:01:22.161460 5371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:46295 every 8 connection(s)
I20260929 03:01:22.162827 5305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:22.166440 5372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:22.171553 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5305
I20260929 03:01:22.171757 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:22.172361 2870 external_mini_cluster.cc:1428] Setting key e596face9f7e5b1574f639d1e86bf4cc
I20260929 03:01:22.176834 5372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e: Bootstrap starting.
I20260929 03:01:22.180545 5372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:22.181823 5372 log.cc:824] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:22.185292 5372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e: No bootstrap required, opened a new log
Sep 29 03:01:22 dist-test-slave-530n krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:22.194147 5372 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } }
I20260929 03:01:22.194674 5372 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:22.194788 5372 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb0741b5600416eae01219fa497548e, State: Initialized, Role: FOLLOWER
I20260929 03:01:22.195525 5372 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } }
I20260929 03:01:22.195765 5372 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:22.195920 5372 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:22.196089 5372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:22.196719 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:22.174374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.2.205.190:46295)
0929 03:01:22.175157 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:22.175169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:22.175216 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:01:22.176974 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:22.177007 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:22.177453 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:22.177804 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:22.179617 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.181048 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.182841 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.183767 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.183914 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:22.188676 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:22.188685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:22.188702 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:22.188798 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:22.191875 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:22.192808 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:22.192816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:22.192824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:22.193024 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:22.193444 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:22.193448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:22.193450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:22.194052 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:22.194277 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:22.194867 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 03:01:22.195367 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":347,"threads_started":1}
I20260929 03:01:22.198585 5372 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } }
I20260929 03:01:22.199110 5372 leader_election.cc:302] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbb0741b5600416eae01219fa497548e; no voters:
I20260929 03:01:22.199718 5372 leader_election.cc:288] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:22.199848 5377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:22.200513 5377 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [term 1 LEADER]: Becoming Leader. State: Replica: fbb0741b5600416eae01219fa497548e, State: Running, Role: LEADER
I20260929 03:01:22.201133 5377 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } }
I20260929 03:01:22.201900 5372 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:22.203738 5379 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb0741b5600416eae01219fa497548e. Latest consensus state: current_term: 1 leader_uuid: "fbb0741b5600416eae01219fa497548e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } } }
I20260929 03:01:22.203770 5378 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbb0741b5600416eae01219fa497548e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb0741b5600416eae01219fa497548e" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46295 } } }
I20260929 03:01:22.204231 5378 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:22.204231 5379 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:22.205143 5384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:22.211521 5384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:22.220095 5384 catalog_manager.cc:1347] Generated new cluster ID: 0b7d45b03ad44635985f52fd6047d03f
I20260929 03:01:22.220204 5384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:22.233703 5384 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:22.235592 5384 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:22.248730 5384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbb0741b5600416eae01219fa497548e: Generated new TSK 0
I20260929 03:01:22.249938 5384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:22.297009 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:22.620937 5398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:22.621243 5398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:22.621369 5398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:22.629067 5398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:22.629190 5398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:22.629340 5398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:22.639276 5398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:22.640646 5398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:22.642020 5398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:22.654805 5406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:22.654871 5403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:22.654868 5404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:22.655782 5398 server_base.cc:1034] running on GCE node
I20260929 03:01:22.656401 5398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:22.657375 5398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:22.658560 5398 hybrid_clock.cc:648] HybridClock initialized: now 1790650882658501 us; error 56 us; skew 500 ppm
I20260929 03:01:22.658857 5398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:22 dist-test-slave-530n krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:22.661859 5398 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:22.663252 5398 webserver.cc:460] Webserver started at http://127.2.205.129:39621/ using document root <none> and password file <none>
I20260929 03:01:22.663918 5398 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:22.664028 5398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:22.664311 5398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:22.667194 5398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "a97d0e0c360b4bc38c9eb53f60a79ece"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "b41190827e6933f501bc51f2c364ced1"
server_key_iv: "4629d8e59e85f62c3e5849018654e749"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.667943 5398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "a97d0e0c360b4bc38c9eb53f60a79ece"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "b41190827e6933f501bc51f2c364ced1"
server_key_iv: "4629d8e59e85f62c3e5849018654e749"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.672703 5398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 03:01:22.675884 5413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:22.677167 5398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:22.677310 5398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "a97d0e0c360b4bc38c9eb53f60a79ece"
format_stamp: "Formatted at 2026-09-29 03:01:22 on dist-test-slave-530n"
server_key: "b41190827e6933f501bc51f2c364ced1"
server_key_iv: "4629d8e59e85f62c3e5849018654e749"
server_key_version: "encryptionkey@0"
I20260929 03:01:22.677487 5398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:22.712374 5398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:22.717597 5398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:22.717907 5398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:22.718921 5398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:22.720309 5398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:22.720386 5398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:22.720507 5398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:22.720557 5398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:22.751085 5398 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:32959
I20260929 03:01:22.751125 5525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:32959 every 8 connection(s)
I20260929 03:01:22.752302 5398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:22.757411 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5398
I20260929 03:01:22.757627 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:22.758376 2870 external_mini_cluster.cc:1428] Setting key 9e3bbaa8544319df2b967bd8e94ee4fb
Sep 29 03:01:22 dist-test-slave-530n krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650882, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:22.772361 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:22.754597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45925 (local address 127.2.205.190:46295)
0929 03:01:22.754986 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:22.754990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:22.755867 (+ 877us) server_negotiation.cc:407] Connection header received
0929 03:01:22.758138 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:22.758146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:22.758267 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:22.758389 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:22.760072 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.760943 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.763143 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.763556 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.763650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:22.767271 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:22.767276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:22.767279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:22.767318 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:22.769591 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:22.770235 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:22.770240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:22.770244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:22.770364 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:22.770744 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:22.770748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:22.770756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:22.771001 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:22.771139 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:22.771792 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 03:01:22.772009 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":201,"threads_started":1}
I20260929 03:01:22.773368 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:22.755046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46295 (local address 127.2.205.129:45925)
0929 03:01:22.755695 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:22.755740 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:22.757733 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:22.758692 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:22.758704 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:22.759317 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:22.759906 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:22.759924 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.761092 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.761097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:22.762998 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:22.763014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:22.763774 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:22.763782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:22.764067 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:22.764736 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:22.764789 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:22.767066 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:22.769712 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:22.769722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:22.769738 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:22.770125 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:22.770463 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:22.770468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:22.770472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:22.770641 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:22.771120 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:22.771128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:22.771447 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 03:01:22.771683 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 03:01:22.772191 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:22.774803 5526 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46295
I20260929 03:01:22.775074 5526 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:22.775792 5526 heartbeater.cc:511] Master 127.2.205.190:46295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:22.777925 5337 ts_manager.cc:194] Registered new tserver with Master: a97d0e0c360b4bc38c9eb53f60a79ece (127.2.205.129:32959)
I20260929 03:01:22.779392 5337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:45925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:22.798079 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:23.114295 5532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:23.114632 5532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:23.114727 5532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:23.122287 5532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:23.122419 5532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:23.122535 5532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:23.132500 5532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:23.133801 5532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:23.135043 5532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:23.149156 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:23.149112 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:23.149189 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:23.150110 5532 server_base.cc:1034] running on GCE node
I20260929 03:01:23.150812 5532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:23.151793 5532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:23.152994 5532 hybrid_clock.cc:648] HybridClock initialized: now 1790650883152940 us; error 50 us; skew 500 ppm
I20260929 03:01:23.153306 5532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:23.159511 5532 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:23.161070 5532 webserver.cc:460] Webserver started at http://127.2.205.130:40565/ using document root <none> and password file <none>
I20260929 03:01:23.161763 5532 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:23.161875 5532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:23.162159 5532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:23.165071 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "2e6144078c0c4b68b7ed398db016c32d"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "bd270870cb90cbec29f3ee5287c38828"
server_key_iv: "4d282e7031878e5e6300cb29306a7d11"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.165822 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "2e6144078c0c4b68b7ed398db016c32d"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "bd270870cb90cbec29f3ee5287c38828"
server_key_iv: "4d282e7031878e5e6300cb29306a7d11"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.170840 5532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 03:01:23.174398 5547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.175814 5532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:23.175966 5532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "2e6144078c0c4b68b7ed398db016c32d"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "bd270870cb90cbec29f3ee5287c38828"
server_key_iv: "4d282e7031878e5e6300cb29306a7d11"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.176156 5532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:23.190012 5532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:23.193800 5532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:23.194052 5532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:23.195080 5532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:23.196346 5532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:23.196421 5532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.196539 5532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:23.196595 5532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.225919 5532 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:42903
I20260929 03:01:23.225955 5659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:42903 every 8 connection(s)
I20260929 03:01:23.227210 5532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:23.235960 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5532
I20260929 03:01:23.236178 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:23.236709 2870 external_mini_cluster.cc:1428] Setting key 970d225ae1bae1c603d9c478ade9a202
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:23.253252 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:23.229743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:50061 (local address 127.2.205.190:46295)
0929 03:01:23.229939 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:23.229946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:23.235080 (+ 5134us) server_negotiation.cc:407] Connection header received
0929 03:01:23.237759 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:23.237765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:23.237863 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:23.237960 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:23.239830 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.240551 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.242692 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.243110 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.243213 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:23.246814 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:23.246821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:23.246826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:23.246865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:23.249997 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:23.250714 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:23.250720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:23.250724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:23.250878 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:23.251404 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:23.251410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:23.251414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:23.251787 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:23.251941 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:23.252650 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 03:01:23.252905 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:01:23.254101 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:23.230317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46295 (local address 127.2.205.130:50061)
0929 03:01:23.234829 (+ 4512us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:23.234899 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:23.237411 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:23.238263 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:23.238274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:23.238830 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:23.239641 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:23.239671 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.240696 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.240701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:23.242535 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:23.242550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.243321 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.243327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:23.243608 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:23.244315 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:23.244337 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:23.246585 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:23.250150 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:23.250157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:23.250170 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:23.250586 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:23.251034 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:23.251039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:23.251044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:23.251273 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:23.251911 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:23.251917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:23.252241 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 03:01:23.252493 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 03:01:23.252931 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4162,"thread_start_us":3935,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:23.255489 5660 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46295
I20260929 03:01:23.255779 5660 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:23.256503 5660 heartbeater.cc:511] Master 127.2.205.190:46295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:23.257943 5337 ts_manager.cc:194] Registered new tserver with Master: 2e6144078c0c4b68b7ed398db016c32d (127.2.205.130:42903)
I20260929 03:01:23.258736 5337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:50061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:23.278462 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:23.594334 5665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:23.594672 5665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:23.594805 5665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:23.602352 5665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:23.602478 5665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:23.602627 5665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:23.612681 5665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:23.614099 5665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:23.615527 5665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:23.629032 5671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:23.629007 5673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:23.629069 5670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:23.629904 5665 server_base.cc:1034] running on GCE node
I20260929 03:01:23.630505 5665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:23.631440 5665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:23.632735 5665 hybrid_clock.cc:648] HybridClock initialized: now 1790650883632670 us; error 53 us; skew 500 ppm
I20260929 03:01:23.633106 5665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:23.636129 5665 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:23.637598 5665 webserver.cc:460] Webserver started at http://127.2.205.131:42955/ using document root <none> and password file <none>
I20260929 03:01:23.638267 5665 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:23.638373 5665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:23.638628 5665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:23.641574 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "bf6e63ed587d43a5ae525d523b8a339f"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "26a1ccb7b439677c66932076654e83d3"
server_key_iv: "f22bb9b7bf6272f60f796b4824e5899a"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.642319 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "bf6e63ed587d43a5ae525d523b8a339f"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "26a1ccb7b439677c66932076654e83d3"
server_key_iv: "f22bb9b7bf6272f60f796b4824e5899a"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.647261 5665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:23.650687 5680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.652056 5665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:01:23.652189 5665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "bf6e63ed587d43a5ae525d523b8a339f"
format_stamp: "Formatted at 2026-09-29 03:01:23 on dist-test-slave-530n"
server_key: "26a1ccb7b439677c66932076654e83d3"
server_key_iv: "f22bb9b7bf6272f60f796b4824e5899a"
server_key_version: "encryptionkey@0"
I20260929 03:01:23.652369 5665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:23.664664 5665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:23.668480 5665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:23.668727 5665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:23.669580 5665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:23.670770 5665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:23.670842 5665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.670946 5665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:23.670994 5665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:23.701128 5665 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:40575
I20260929 03:01:23.701172 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:40575 every 8 connection(s)
I20260929 03:01:23.702555 5665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:23.705365 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5665
I20260929 03:01:23.705596 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:23.706112 2870 external_mini_cluster.cc:1428] Setting key 0c8be69d9e134d564cb90a5c4f64a9f9
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650883, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:23.725078 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:23.705240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:39279 (local address 127.2.205.190:46295)
0929 03:01:23.705414 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:23.705419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:23.707186 (+ 1767us) server_negotiation.cc:407] Connection header received
0929 03:01:23.709546 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:23.709551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:23.709635 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:23.709716 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:23.711342 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.712103 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.713718 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.714121 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.714241 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:23.718765 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:23.718770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:23.718773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:23.718812 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:23.721808 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:23.722613 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:23.722619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:23.722623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:23.722782 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:23.723238 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:23.723244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:23.723247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:23.723549 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:23.723679 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:23.724456 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 03:01:23.724690 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:01:23.726117 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:23.706077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46295 (local address 127.2.205.131:39279)
0929 03:01:23.707019 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:23.707068 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:23.709277 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:23.709980 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:23.709989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:23.710561 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:23.711175 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:23.711196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.712257 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.712261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:23.713578 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:23.713591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:23.714346 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:23.714353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:23.714606 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:23.715381 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:23.715405 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:23.718497 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:23.721967 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:23.721977 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:23.721995 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:23.722481 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:23.722915 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:23.722921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:23.722924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:23.723121 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:23.723672 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:23.723681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:23.724064 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 03:01:23.724314 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 03:01:23.724968 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":333,"threads_started":1}
I20260929 03:01:23.727507 5793 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46295
I20260929 03:01:23.727800 5793 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:23.728524 5793 heartbeater.cc:511] Master 127.2.205.190:46295 requested a full tablet report, sending...
I20260929 03:01:23.729980 5337 ts_manager.cc:194] Registered new tserver with Master: bf6e63ed587d43a5ae525d523b8a339f (127.2.205.131:40575)
I20260929 03:01:23.730849 5337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:39279
I20260929 03:01:23.734239 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:01:23.762564 5528 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:23.760361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.2.205.190:46295)
0929 03:01:23.760520 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:23.760524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:23.761018 (+ 494us) server_negotiation.cc:407] Connection header received
0929 03:01:23.761161 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:23.761165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:23.761225 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:23.761312 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:23.762337 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48748: BlockingRecv error: recv got EOF from 127.0.0.1:48748 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:01:23.765712 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5398
W20260929 03:01:23.768205 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5398: Not authorized: Client connection negotiation failed: client connection to 127.2.205.129:32959: server requires authentication, but client does not have Kerberos credentials available
I20260929 03:01:23.768352 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5398
I20260929 03:01:23.780488 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5532
W20260929 03:01:23.783087 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5532: Not authorized: Client connection negotiation failed: client connection to 127.2.205.130:42903: server requires authentication, but client does not have Kerberos credentials available
I20260929 03:01:23.783279 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5532
I20260929 03:01:23.795308 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5665
W20260929 03:01:23.798157 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5665: Not authorized: Client connection negotiation failed: client connection to 127.2.205.131:40575: server requires authentication, but client does not have Kerberos credentials available
I20260929 03:01:23.798295 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5665
I20260929 03:01:23.810575 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5305
I20260929 03:01:23.811839 5367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:23.913482 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5305
2026-09-29T03:01:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3344 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5810](info): set up 2 sockets
Sep 29 03:01:23 dist-test-slave-530n krb5kdc[5810](info): commencing operation
krb5kdc: starting...
W20260929 03:01:24.905051 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:25 dist-test-slave-530n krb5kdc[5810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:25.098184 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36349
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:25.422008 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:25.422396 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:25.422492 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:25.430094 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:25.430220 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:25.430274 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:25.430330 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:25.430377 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:25.440809 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:25.442245 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:25.443605 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:25.455928 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:25.456005 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:25.455957 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:25.456799 5823 server_base.cc:1034] running on GCE node
I20260929 03:01:25.457592 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:25.459077 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:25.460295 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790650885460238 us; error 44 us; skew 500 ppm
I20260929 03:01:25.460629 5823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:25 dist-test-slave-530n krb5kdc[5810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650885, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:25.463624 5823 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:25.465122 5823 webserver.cc:460] Webserver started at http://127.2.205.190:44331/ using document root <none> and password file <none>
I20260929 03:01:25.465792 5823 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:25.465905 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:25.466188 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:25.469134 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "0a2cdc2bb3394e818fb036ea917b67fa"
format_stamp: "Formatted at 2026-09-29 03:01:25 on dist-test-slave-530n"
server_key: "e5bd9bef7b205846587ca1bb62171a03"
server_key_iv: "2f798f846abb3cee15a11072fd960c16"
server_key_version: "encryptionkey@0"
I20260929 03:01:25.469883 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "0a2cdc2bb3394e818fb036ea917b67fa"
format_stamp: "Formatted at 2026-09-29 03:01:25 on dist-test-slave-530n"
server_key: "e5bd9bef7b205846587ca1bb62171a03"
server_key_iv: "2f798f846abb3cee15a11072fd960c16"
server_key_version: "encryptionkey@0"
I20260929 03:01:25.475852 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 03:01:25.479509 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:25.481030 5823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:25.481194 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "0a2cdc2bb3394e818fb036ea917b67fa"
format_stamp: "Formatted at 2026-09-29 03:01:25 on dist-test-slave-530n"
server_key: "e5bd9bef7b205846587ca1bb62171a03"
server_key_iv: "2f798f846abb3cee15a11072fd960c16"
server_key_version: "encryptionkey@0"
I20260929 03:01:25.481388 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:25.509778 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:25.514175 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:25.514456 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:25.532371 5823 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36349
I20260929 03:01:25.532428 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36349 every 8 connection(s)
I20260929 03:01:25.533874 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:25.535698 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5823
I20260929 03:01:25.535866 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:25.536490 2870 external_mini_cluster.cc:1428] Setting key cf97b1c5510a726c72568b91483d3029
I20260929 03:01:25.538146 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:01:25 dist-test-slave-530n krb5kdc[5810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:25.551708 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa: Bootstrap starting.
I20260929 03:01:25.555337 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:25.556584 5890 log.cc:824] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:25.559870 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa: No bootstrap required, opened a new log
I20260929 03:01:25.560263 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:25.538336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.2.205.190:36349)
0929 03:01:25.539113 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:25.539125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:25.539165 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:01:25.540808 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:25.540830 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:25.541296 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:25.541638 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:25.544387 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:25.545732 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:25.547454 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:25.548161 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:25.548301 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:25.552985 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:25.552992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:25.553006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:25.553046 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:25.555861 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:25.556637 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:25.556644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:25.556649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:25.556858 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:25.557273 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:25.557278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:25.557285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:25.558012 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:25.558220 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:25.558754 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 03:01:25.559142 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
I20260929 03:01:25.564110 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:01:25.569208 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } }
I20260929 03:01:25.569681 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:25.569782 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2cdc2bb3394e818fb036ea917b67fa, State: Initialized, Role: FOLLOWER
I20260929 03:01:25.570436 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } }
I20260929 03:01:25.570691 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:25.570808 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:25.570976 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:25.574090 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } }
I20260929 03:01:25.574685 5890 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2cdc2bb3394e818fb036ea917b67fa; no voters:
I20260929 03:01:25.576515 5890 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:25.576649 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:25.577991 5896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [term 1 LEADER]: Becoming Leader. State: Replica: 0a2cdc2bb3394e818fb036ea917b67fa, State: Running, Role: LEADER
I20260929 03:01:25.578689 5896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } }
I20260929 03:01:25.579116 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:25.582046 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2cdc2bb3394e818fb036ea917b67fa. Latest consensus state: current_term: 1 leader_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } } }
I20260929 03:01:25.582616 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:25.586017 5903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:25.585915 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2cdc2bb3394e818fb036ea917b67fa" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36349 } } }
I20260929 03:01:25.586287 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:25.591763 5903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:25.601783 5903 catalog_manager.cc:1347] Generated new cluster ID: 7bdca206c9604b868ce65494b66d3440
I20260929 03:01:25.601924 5903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:25.610059 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:25.601794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.2.205.190:36349)
0929 03:01:25.601967 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:25.601973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:25.602760 (+ 787us) server_negotiation.cc:407] Connection header received
0929 03:01:25.602971 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:25.602977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:25.603069 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:25.603169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:25.604390 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:25.605227 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:25.606541 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:25.606974 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:25.607102 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:25.608329 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:25.608336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:25.608340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:25.609207 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:25.609276 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:25.609312 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:25.609412 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:25.609565 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:01:25.609681 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:01:25.615741 5903 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:25.617501 5903 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:25.627775 5903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2cdc2bb3394e818fb036ea917b67fa: Generated new TSK 0
I20260929 03:01:25.628808 5903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:01:25.637473 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5823
I20260929 03:01:25.638887 5885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:25.731396 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5823
2026-09-29T03:01:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1813 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:27 dist-test-slave-530n krb5kdc[5924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:27 dist-test-slave-530n krb5kdc[5924](info): set up 2 sockets
Sep 29 03:01:27 dist-test-slave-530n krb5kdc[5924](info): commencing operation
krb5kdc: starting...
W20260929 03:01:28.173166 2870 mini_kdc.cc:121] Time spent starting KDC: real 2.408s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:28 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:28.358265 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46561
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:28.678095 5937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:28.678402 5937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:28.678535 5937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:28.685756 5937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:28.685890 5937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:28.685974 5937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:28.686048 5937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:28.686105 5937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:28.696174 5937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:28.697629 5937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:28.698979 5937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:28.710265 5942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:28.710214 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:28.710412 5945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:28.711122 5937 server_base.cc:1034] running on GCE node
I20260929 03:01:28.711916 5937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:28.713475 5937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:28.714699 5937 hybrid_clock.cc:648] HybridClock initialized: now 1790650888714623 us; error 68 us; skew 500 ppm
I20260929 03:01:28.715056 5937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:28 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:28.724641 5937 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:28.726367 5937 webserver.cc:460] Webserver started at http://127.2.205.190:45587/ using document root <none> and password file <none>
I20260929 03:01:28.727120 5937 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:28.727236 5937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:28.727522 5937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:28.730727 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "f921ccc4f77a46e4bb540e7d8202043a"
format_stamp: "Formatted at 2026-09-29 03:01:28 on dist-test-slave-530n"
server_key: "fad8aaa2a0b8c361ee472243dcf55bac"
server_key_iv: "d958d945dd27db6a1289273aa0ee17d8"
server_key_version: "encryptionkey@0"
I20260929 03:01:28.731542 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "f921ccc4f77a46e4bb540e7d8202043a"
format_stamp: "Formatted at 2026-09-29 03:01:28 on dist-test-slave-530n"
server_key: "fad8aaa2a0b8c361ee472243dcf55bac"
server_key_iv: "d958d945dd27db6a1289273aa0ee17d8"
server_key_version: "encryptionkey@0"
I20260929 03:01:28.738305 5937 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 03:01:28.741834 5952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:28.743310 5937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:01:28.743475 5937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "f921ccc4f77a46e4bb540e7d8202043a"
format_stamp: "Formatted at 2026-09-29 03:01:28 on dist-test-slave-530n"
server_key: "fad8aaa2a0b8c361ee472243dcf55bac"
server_key_iv: "d958d945dd27db6a1289273aa0ee17d8"
server_key_version: "encryptionkey@0"
I20260929 03:01:28.743669 5937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:28.763335 5937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:28.767370 5937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:28.767611 5937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:28.784325 5937 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:46561
I20260929 03:01:28.784356 6003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:46561 every 8 connection(s)
I20260929 03:01:28.785722 5937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:28.789702 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:28.795918 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 5937
I20260929 03:01:28.796149 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:28.796996 2870 external_mini_cluster.cc:1428] Setting key d0f280888a92e94bc46d0869f6df7186
I20260929 03:01:28.801371 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a: Bootstrap starting.
I20260929 03:01:28.805131 6004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:28.806213 6004 log.cc:824] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:28.809456 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a: No bootstrap required, opened a new log
Sep 29 03:01:28 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:28.818557 6004 raft_consensus.cc:348] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } }
I20260929 03:01:28.818959 6004 raft_consensus.cc:374] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:28.819072 6004 raft_consensus.cc:729] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f921ccc4f77a46e4bb540e7d8202043a, State: Initialized, Role: FOLLOWER
I20260929 03:01:28.819854 6004 consensus_queue.cc:260] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } }
I20260929 03:01:28.820151 6004 raft_consensus.cc:388] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:28.820313 6004 raft_consensus.cc:482] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:28.820504 6004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:28.821658 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:28.799293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.2.205.190:46561)
0929 03:01:28.800143 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:28.800157 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:28.800203 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:01:28.801981 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:28.802008 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:28.802456 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:28.802823 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:28.804556 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:28.806038 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:28.807530 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:28.808213 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:28.808368 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:28.813468 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:28.813478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:28.813499 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:28.813554 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:28.816870 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:28.817832 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:28.817841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:28.817848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:28.818049 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:28.818485 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:28.818488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:28.818490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:28.819159 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:28.819418 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:28.819887 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 03:01:28.820346 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":373,"threads_started":1}
I20260929 03:01:28.823071 6004 raft_consensus.cc:504] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } }
I20260929 03:01:28.823704 6004 leader_election.cc:302] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f921ccc4f77a46e4bb540e7d8202043a; no voters:
I20260929 03:01:28.824389 6004 leader_election.cc:288] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:28.824514 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:28.825922 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [term 1 LEADER]: Becoming Leader. State: Replica: f921ccc4f77a46e4bb540e7d8202043a, State: Running, Role: LEADER
I20260929 03:01:28.826453 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } }
I20260929 03:01:28.826695 6004 sys_catalog.cc:564] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:28.829214 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f921ccc4f77a46e4bb540e7d8202043a. Latest consensus state: current_term: 1 leader_uuid: "f921ccc4f77a46e4bb540e7d8202043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } } }
I20260929 03:01:28.829311 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f921ccc4f77a46e4bb540e7d8202043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f921ccc4f77a46e4bb540e7d8202043a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46561 } } }
I20260929 03:01:28.829573 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:28.829675 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:28.831243 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:28.837051 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:28.847067 6017 catalog_manager.cc:1347] Generated new cluster ID: a3b41d35242944aabf68cae180b91acc
I20260929 03:01:28.847193 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:28.871115 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:28.872699 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:28.881434 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f921ccc4f77a46e4bb540e7d8202043a: Generated new TSK 0
I20260929 03:01:28.882484 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:28.936787 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:29.264593 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:29.264879 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:29.264998 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:29.272493 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:29.272585 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:29.272737 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:29.282119 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:29.283463 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:29.284811 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:29.298012 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:29.298049 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:29.298070 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:29.298818 6030 server_base.cc:1034] running on GCE node
I20260929 03:01:29.299484 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:29.300482 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:29.301699 6030 hybrid_clock.cc:648] HybridClock initialized: now 1790650889301637 us; error 55 us; skew 500 ppm
I20260929 03:01:29.302057 6030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:29 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:29.305261 6030 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:29.306777 6030 webserver.cc:460] Webserver started at http://127.2.205.129:39751/ using document root <none> and password file <none>
I20260929 03:01:29.307467 6030 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:29.307596 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:29.307912 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:29.310958 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "770602fee67042c0a4068a7151ede6fb"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "652c5559052ffbebe80cebec782dbb61"
server_key_iv: "e175555cc3eaa7d6a33d953327e13237"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.311726 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "770602fee67042c0a4068a7151ede6fb"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "652c5559052ffbebe80cebec782dbb61"
server_key_iv: "e175555cc3eaa7d6a33d953327e13237"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.317078 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 03:01:29.320946 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.322772 6030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:01:29.322933 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "770602fee67042c0a4068a7151ede6fb"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "652c5559052ffbebe80cebec782dbb61"
server_key_iv: "e175555cc3eaa7d6a33d953327e13237"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.323124 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:29.343629 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:29.348042 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:29.348333 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:29.349385 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:29.350656 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:29.350723 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.350842 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:29.350891 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.379179 6030 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37363
I20260929 03:01:29.379222 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37363 every 8 connection(s)
I20260929 03:01:29.380486 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:29.386121 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6030
I20260929 03:01:29.386327 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:29.386950 2870 external_mini_cluster.cc:1428] Setting key 4f067f732f05d1c1c226c1c65207914b
Sep 29 03:01:29 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:29.401594 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:29.383025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:33307 (local address 127.2.205.190:46561)
0929 03:01:29.383462 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:29.383469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:29.384574 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 03:01:29.386454 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:29.386460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:29.386545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:29.386639 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:29.388357 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.389246 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.390817 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.391221 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.391332 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:29.395090 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:29.395097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:29.395102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:29.395144 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:29.398201 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:29.398996 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:29.399001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:29.399005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:29.399151 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:29.399562 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:29.399567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:29.399570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:29.399910 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:29.400045 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:29.401111 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0929 03:01:29.401320 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260929 03:01:29.403174 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:29.383608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46561 (local address 127.2.205.129:33307)
0929 03:01:29.384406 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:29.384452 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:29.386147 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:29.386901 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:29.386910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:29.387533 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:29.388186 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:29.388204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.389412 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.389416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:29.390667 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:29.390679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.391458 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.391466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:29.391690 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:29.392389 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:29.392419 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:29.394830 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:29.398381 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:29.398389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:29.398404 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:29.398879 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:29.399261 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:29.399264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:29.399267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:29.399459 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:29.400124 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:29.400133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:29.400651 (+ 518us) client_negotiation.cc:769] Sending connection context
0929 03:01:29.401014 (+ 363us) client_negotiation.cc:240] Negotiation successful
0929 03:01:29.401657 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:29.404939 6158 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46561
I20260929 03:01:29.405222 6158 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:29.406006 6158 heartbeater.cc:511] Master 127.2.205.190:46561 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:29.408586 5969 ts_manager.cc:194] Registered new tserver with Master: 770602fee67042c0a4068a7151ede6fb (127.2.205.129:37363)
I20260929 03:01:29.410580 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:33307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:29.432189 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:29.750813 6164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:29.751101 6164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:29.751194 6164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:29.758594 6164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:29.758715 6164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:29.758831 6164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:29.768566 6164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:29.769840 6164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:29.771108 6164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:29.784165 6170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:29.784345 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:29.784638 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:29.785862 6164 server_base.cc:1034] running on GCE node
I20260929 03:01:29.786491 6164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:29.787479 6164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:29.788686 6164 hybrid_clock.cc:648] HybridClock initialized: now 1790650889788625 us; error 60 us; skew 500 ppm
I20260929 03:01:29.789029 6164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:29 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:29.792109 6164 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:29.793790 6164 webserver.cc:460] Webserver started at http://127.2.205.130:38913/ using document root <none> and password file <none>
I20260929 03:01:29.794488 6164 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:29.794600 6164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:29.794919 6164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:29.797997 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "14d29459f61b4e2a9cd4c4027ad4d1bf"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "5772a38c866667d30ea4be13463caa19"
server_key_iv: "0dfe6a675a22207a2c7582c712949768"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.798871 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "14d29459f61b4e2a9cd4c4027ad4d1bf"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "5772a38c866667d30ea4be13463caa19"
server_key_iv: "0dfe6a675a22207a2c7582c712949768"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.804190 6164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:01:29.807919 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.809564 6164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:01:29.809747 6164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "14d29459f61b4e2a9cd4c4027ad4d1bf"
format_stamp: "Formatted at 2026-09-29 03:01:29 on dist-test-slave-530n"
server_key: "5772a38c866667d30ea4be13463caa19"
server_key_iv: "0dfe6a675a22207a2c7582c712949768"
server_key_version: "encryptionkey@0"
I20260929 03:01:29.809962 6164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:29.830492 6164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:29.834879 6164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:29.835153 6164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:29.836151 6164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:29.837673 6164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:29.837756 6164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.837873 6164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:29.837924 6164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:29.867540 6164 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:45185
I20260929 03:01:29.867581 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:45185 every 8 connection(s)
I20260929 03:01:29.868795 6164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:29.869897 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6164
I20260929 03:01:29.870074 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:29.870546 2870 external_mini_cluster.cc:1428] Setting key 7d5889a6ac4c4df9248e94396c168033
Sep 29 03:01:29 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:29.889832 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:29.871445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:41421 (local address 127.2.205.190:46561)
0929 03:01:29.871643 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:29.871649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:29.872918 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 03:01:29.874669 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:29.874675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:29.874751 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:29.874854 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:29.876268 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.877151 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.879402 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.879852 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.879955 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:29.884054 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:29.884060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:29.884065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:29.884109 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:29.886550 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:29.887436 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:29.887440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:29.887442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:29.887584 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:29.888020 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:29.888024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:29.888028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:29.888322 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:29.888457 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:29.889232 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 03:01:29.889484 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:29.891086 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:29.871943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46561 (local address 127.2.205.130:41421)
0929 03:01:29.872769 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:29.872816 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:29.874416 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:29.875046 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:29.875053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:29.875566 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:29.876110 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:29.876129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.877313 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.877318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:29.879241 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:29.879256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:29.880068 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:29.880075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:29.880355 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:29.881087 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:29.881121 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:29.883794 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:29.886691 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:29.886700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:29.886718 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:29.887314 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:29.887696 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:29.887702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:29.887706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:29.887913 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:29.888448 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:29.888456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:29.888879 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 03:01:29.889137 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 03:01:29.889764 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260929 03:01:29.892580 6292 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46561
I20260929 03:01:29.892926 6292 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:29.893775 6292 heartbeater.cc:511] Master 127.2.205.190:46561 requested a full tablet report, sending...
I20260929 03:01:29.895346 5969 ts_manager.cc:194] Registered new tserver with Master: 14d29459f61b4e2a9cd4c4027ad4d1bf (127.2.205.130:45185)
I20260929 03:01:29.896145 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:41421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:29.912515 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:30.227648 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:30.228003 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:30.228149 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:30.235678 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:30.235803 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:30.235929 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:30.245919 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:30.247298 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:30.248603 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:30.260869 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:30.260993 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:30.260840 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:30.261806 6297 server_base.cc:1034] running on GCE node
I20260929 03:01:30.262516 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:30.263604 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:30.264789 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790650890264707 us; error 50 us; skew 500 ppm
I20260929 03:01:30.265148 6297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:30.268620 6297 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:30.270028 6297 webserver.cc:460] Webserver started at http://127.2.205.131:37643/ using document root <none> and password file <none>
I20260929 03:01:30.270680 6297 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:30.270804 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:30.271090 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:30.274104 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "6eb20dcfef0a490186177e50c24b82cf"
format_stamp: "Formatted at 2026-09-29 03:01:30 on dist-test-slave-530n"
server_key: "ca098a8b5a3d2423a411e15e3235c164"
server_key_iv: "e7f12b9e4763ec9fc0cff7e8a0c690f8"
server_key_version: "encryptionkey@0"
I20260929 03:01:30.274850 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "6eb20dcfef0a490186177e50c24b82cf"
format_stamp: "Formatted at 2026-09-29 03:01:30 on dist-test-slave-530n"
server_key: "ca098a8b5a3d2423a411e15e3235c164"
server_key_iv: "e7f12b9e4763ec9fc0cff7e8a0c690f8"
server_key_version: "encryptionkey@0"
I20260929 03:01:30.280133 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 03:01:30.284293 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:30.286021 6297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:30.286223 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "6eb20dcfef0a490186177e50c24b82cf"
format_stamp: "Formatted at 2026-09-29 03:01:30 on dist-test-slave-530n"
server_key: "ca098a8b5a3d2423a411e15e3235c164"
server_key_iv: "e7f12b9e4763ec9fc0cff7e8a0c690f8"
server_key_version: "encryptionkey@0"
I20260929 03:01:30.286464 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:30.311123 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:30.316099 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:30.316372 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:30.317409 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:30.318856 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:30.318938 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:30.319043 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:30.319093 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:30.348326 6297 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:34063
I20260929 03:01:30.348364 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:34063 every 8 connection(s)
I20260929 03:01:30.349689 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:30.360015 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6297
I20260929 03:01:30.360165 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:30.360616 2870 external_mini_cluster.cc:1428] Setting key e023a0a170170e098e3bcb74181feb4e
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:30.368120 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.352150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:44339 (local address 127.2.205.190:46561)
0929 03:01:30.352354 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:30.352358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:30.353520 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 03:01:30.355292 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:30.355307 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:30.355385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:30.355469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:30.356912 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.357669 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.359150 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.359567 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.359672 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.363257 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:30.363262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:30.363265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:30.363298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:30.365660 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.366275 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.366279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.366282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.366411 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.366747 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.366750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.366752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.367019 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:30.367127 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:30.367708 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 03:01:30.367865 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:01:30.369067 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.352661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46561 (local address 127.2.205.131:44339)
0929 03:01:30.353368 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:30.353407 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:30.355025 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:30.355696 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:30.355705 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:30.356193 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:30.356728 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:30.356767 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.357819 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.357822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:30.359021 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:30.359033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.359789 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.359795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:30.360010 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.360626 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:30.360647 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:30.363066 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:30.365771 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:30.365778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:30.365789 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:30.366166 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:30.366497 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:30.366500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:30.366502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:30.366664 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:30.367107 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:30.367113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:30.367407 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 03:01:30.367614 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 03:01:30.367953 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":254,"threads_started":1}
I20260929 03:01:30.370335 6425 heartbeater.cc:347] Connected to a master server at 127.2.205.190:46561
I20260929 03:01:30.370596 6425 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:30.371279 6425 heartbeater.cc:511] Master 127.2.205.190:46561 requested a full tablet report, sending...
I20260929 03:01:30.372589 5969 ts_manager.cc:194] Registered new tserver with Master: 6eb20dcfef0a490186177e50c24b82cf (127.2.205.131:34063)
I20260929 03:01:30.373381 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:44339
I20260929 03:01:30.375537 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260929 03:01:30.414214 6158 heartbeater.cc:503] Master 127.2.205.190:46561 was elected leader, sending a full tablet report...
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:30.421643 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.407256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.2.205.190:46561)
0929 03:01:30.407450 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:30.407457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:30.407874 (+ 417us) server_negotiation.cc:407] Connection header received
0929 03:01:30.408054 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:30.408058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:30.408124 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:30.408210 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:30.409658 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.410395 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.411658 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.412107 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.412221 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.416477 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:30.416483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:30.416486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:30.416519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:30.418967 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.419645 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.419652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.419655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.419789 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.420161 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.420166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.420168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.420522 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:30.420811 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:30.421107 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:01:30.421335 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:01:30.422156 5969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33256
I20260929 03:01:30.424144 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6030
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:30.440277 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.425192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.2.205.129:37363)
0929 03:01:30.425827 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:30.425837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:30.425866 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:01:30.425928 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:30.425935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:30.426159 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:30.426387 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:30.427583 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.428622 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.429852 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.430498 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.430614 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.434441 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:30.434455 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:30.434472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:30.434513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:30.437423 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.438177 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.438185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.438187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.438329 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.438708 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.438712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.438714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.439072 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:30.439385 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:30.439660 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 03:01:30.439923 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":342,"threads_started":1}
W20260929 03:01:30.441650 6153 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58640
W20260929 03:01:30.442827 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:30.443033 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6030
I20260929 03:01:30.455235 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6164
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:30.471012 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.456148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.2.205.130:45185)
0929 03:01:30.456677 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:30.456684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:30.456709 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:01:30.456792 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:30.456798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:30.457013 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:30.457144 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:30.458667 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.459668 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.460911 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.461440 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.461544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.465495 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:30.465507 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:30.465521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:30.465562 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:30.468224 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.469007 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.469015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.469017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.469143 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.469522 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.469525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.469526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.469840 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:30.470078 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:30.470425 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 03:01:30.470686 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":282,"threads_started":1}
W20260929 03:01:30.472203 6287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57518
W20260929 03:01:30.473273 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:30.473412 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6164
I20260929 03:01:30.485440 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6297
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:30.503086 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:30.486471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.2.205.131:34063)
0929 03:01:30.487127 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:30.487134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:30.487164 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:01:30.487225 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:30.487231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:30.487460 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:30.487590 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:30.488990 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.490070 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.491444 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:30.492147 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:30.492272 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:30.496960 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:30.496983 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:30.497000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:30.497052 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:30.499922 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.500674 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.500683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.500686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.500961 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:30.501413 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:30.501417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:30.501419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:30.501834 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:30.502078 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:30.502479 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 03:01:30.502708 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":330,"threads_started":1}
W20260929 03:01:30.504468 6420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58974
W20260929 03:01:30.505681 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6297: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:01:30.505868 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6297
I20260929 03:01:30.517861 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 5937
I20260929 03:01:30.519028 5999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:30.621852 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 5937
2026-09-29T03:01:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4894 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[6442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[6442](info): set up 2 sockets
Sep 29 03:01:30 dist-test-slave-530n krb5kdc[6442](info): commencing operation
krb5kdc: starting...
W20260929 03:01:31.621122 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:31 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:31.807664 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36501
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:32.126040 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:32.126354 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:32.126446 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:32.133889 6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:32.134009 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:32.134053 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:32.134109 6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:32.134151 6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:32.144701 6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:32.146111 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:32.147418 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:32.158779 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:32.158746 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:32.158731 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:32.159358 6455 server_base.cc:1034] running on GCE node
I20260929 03:01:32.160228 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:32.161721 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:32.162940 6455 hybrid_clock.cc:648] HybridClock initialized: now 1790650892162895 us; error 35 us; skew 500 ppm
I20260929 03:01:32.163254 6455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:32 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:32.166283 6455 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:32.167774 6455 webserver.cc:460] Webserver started at http://127.2.205.190:43313/ using document root <none> and password file <none>
I20260929 03:01:32.168426 6455 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:32.168522 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:32.168856 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:32.171749 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "bbf93433694d43bba3653d776ffdab04"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "530826fa05d934d1efe994e8342e9f90"
server_key_iv: "888bfce3ef02b9e7eb3f4fe50b1ccb93"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.172487 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "bbf93433694d43bba3653d776ffdab04"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "530826fa05d934d1efe994e8342e9f90"
server_key_iv: "888bfce3ef02b9e7eb3f4fe50b1ccb93"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.177759 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:32.181425 6470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:32.182920 6455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:32.183102 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "bbf93433694d43bba3653d776ffdab04"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "530826fa05d934d1efe994e8342e9f90"
server_key_iv: "888bfce3ef02b9e7eb3f4fe50b1ccb93"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.183300 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:32.197876 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:32.202379 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:32.202677 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:32.220571 6455 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36501
I20260929 03:01:32.220589 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36501 every 8 connection(s)
I20260929 03:01:32.222149 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:32.225555 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6455
I20260929 03:01:32.225754 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:32.226109 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:32.226367 2870 external_mini_cluster.cc:1428] Setting key 79220cd02ff31efbc5c3bec21e04b5ba
I20260929 03:01:32.239781 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04: Bootstrap starting.
Sep 29 03:01:32 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:32.243641 6522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:32.245076 6522 log.cc:824] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:32.248631 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04: No bootstrap required, opened a new log
I20260929 03:01:32.249879 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:32.228285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.2.205.190:36501)
0929 03:01:32.229112 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:32.229127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:32.229181 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:01:32.230850 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:32.230879 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:32.231402 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:32.231849 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:32.233183 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.235088 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.236381 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.237486 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.237682 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:32.241394 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:32.241405 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:32.241423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:32.241486 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:32.245285 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:32.246189 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:32.246196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:32.246202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:32.246386 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:32.246769 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:32.246774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:32.246777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:32.247466 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:32.247750 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:32.248039 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:01:32.248494 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":307,"threads_started":1}
I20260929 03:01:32.257081 6522 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } }
I20260929 03:01:32.257522 6522 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:32.257615 6522 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf93433694d43bba3653d776ffdab04, State: Initialized, Role: FOLLOWER
I20260929 03:01:32.258152 6522 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } }
I20260929 03:01:32.258370 6522 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:32.258495 6522 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:32.258646 6522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:32.260987 6522 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } }
I20260929 03:01:32.261472 6522 leader_election.cc:302] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf93433694d43bba3653d776ffdab04; no voters:
I20260929 03:01:32.262022 6522 leader_election.cc:288] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:32.262138 6527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:32.262744 6527 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [term 1 LEADER]: Becoming Leader. State: Replica: bbf93433694d43bba3653d776ffdab04, State: Running, Role: LEADER
I20260929 03:01:32.263208 6527 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } }
I20260929 03:01:32.263952 6522 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:32.265695 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf93433694d43bba3653d776ffdab04. Latest consensus state: current_term: 1 leader_uuid: "bbf93433694d43bba3653d776ffdab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } } }
I20260929 03:01:32.266054 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:32.266698 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf93433694d43bba3653d776ffdab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf93433694d43bba3653d776ffdab04" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36501 } } }
I20260929 03:01:32.267016 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:32.268098 6532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:32.273344 6532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:32.282760 6532 catalog_manager.cc:1347] Generated new cluster ID: e70d41140b9e4790894f03f02cb93d10
I20260929 03:01:32.282899 6532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:32.296604 6532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:32.298154 6532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:32.306476 6532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf93433694d43bba3653d776ffdab04: Generated new TSK 0
I20260929 03:01:32.307391 6532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:32.360565 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:32.687880 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:32.688176 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:32.688310 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:32.696034 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:32.696182 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:32.696341 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:32.706403 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:32.707744 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:32.709160 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:32.722438 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:32.722471 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:32.722438 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:32.723659 6548 server_base.cc:1034] running on GCE node
I20260929 03:01:32.724376 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:32.725505 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:32.726748 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790650892726681 us; error 55 us; skew 500 ppm
I20260929 03:01:32.727083 6548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:32 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:32.730400 6548 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:32.731927 6548 webserver.cc:460] Webserver started at http://127.2.205.129:43149/ using document root <none> and password file <none>
I20260929 03:01:32.732635 6548 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:32.732805 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:32.733184 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:32.736366 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "350107ac02f941cc888a6300883d1fbc"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "9a6865414ce2f160af352e4e5afa3273"
server_key_iv: "842ca9e9188d785dcd4a7d9a51e3e403"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.737203 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "350107ac02f941cc888a6300883d1fbc"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "9a6865414ce2f160af352e4e5afa3273"
server_key_iv: "842ca9e9188d785dcd4a7d9a51e3e403"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.742641 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:32.746567 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:32.748898 6548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:01:32.749038 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "350107ac02f941cc888a6300883d1fbc"
format_stamp: "Formatted at 2026-09-29 03:01:32 on dist-test-slave-530n"
server_key: "9a6865414ce2f160af352e4e5afa3273"
server_key_iv: "842ca9e9188d785dcd4a7d9a51e3e403"
server_key_version: "encryptionkey@0"
I20260929 03:01:32.749217 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:32.786954 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:32.791538 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:32.791844 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:32.792986 6548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:32.794525 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:32.794602 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:32.794710 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:32.794772 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:32.826509 6548 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:46085
I20260929 03:01:32.826555 6675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:46085 every 8 connection(s)
I20260929 03:01:32.827842 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:32.831326 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6548
I20260929 03:01:32.831562 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:32.832350 2870 external_mini_cluster.cc:1428] Setting key b0424f6b66c8db4a851f046470d01859
Sep 29 03:01:32 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650892, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:32.850666 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:32.830363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:39217 (local address 127.2.205.190:36501)
0929 03:01:32.830794 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:32.830800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:32.831965 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 03:01:32.834239 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:32.834245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:32.834325 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:32.834461 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:32.836693 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.837486 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.839805 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.840377 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.840519 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:32.844427 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:32.844433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:32.844436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:32.844472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:32.847137 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:32.848126 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:32.848132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:32.848135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:32.848298 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:32.848837 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:32.848841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:32.848843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:32.849114 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:32.849252 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:32.850199 (+ 947us) server_negotiation.cc:299] Negotiation successful
0929 03:01:32.850410 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":199,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:32.851830 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:32.830967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36501 (local address 127.2.205.129:39217)
0929 03:01:32.831758 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:32.831827 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:32.833818 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:32.834823 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:32.834836 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:32.835587 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:32.836475 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:32.836503 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.837673 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.837678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:32.839557 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:32.839573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:32.840622 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:32.840627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:32.840904 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:32.841611 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:32.841637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:32.844116 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:32.847303 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:32.847314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:32.847345 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:32.847990 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:32.848410 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:32.848415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:32.848419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:32.848672 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:32.849248 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:32.849257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:32.849739 (+ 482us) client_negotiation.cc:769] Sending connection context
0929 03:01:32.850077 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 03:01:32.850657 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:32.853542 6676 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36501
I20260929 03:01:32.853899 6676 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:32.854748 6676 heartbeater.cc:511] Master 127.2.205.190:36501 requested a full tablet report, sending...
I20260929 03:01:32.857125 6487 ts_manager.cc:194] Registered new tserver with Master: 350107ac02f941cc888a6300883d1fbc (127.2.205.129:46085)
I20260929 03:01:32.858982 6487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:39217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:32.875710 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:33.195236 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:33.195574 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:33.195720 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:33.203181 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:33.203310 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:33.203470 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:33.213567 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:33.215093 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:33.216558 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:33.232044 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:33.236297 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:33.236711 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:33.237257 6682 server_base.cc:1034] running on GCE node
I20260929 03:01:33.238029 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:33.239208 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:33.240432 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790650893240368 us; error 57 us; skew 500 ppm
I20260929 03:01:33.240768 6682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:33.243944 6682 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:33.245430 6682 webserver.cc:460] Webserver started at http://127.2.205.130:46241/ using document root <none> and password file <none>
I20260929 03:01:33.246165 6682 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:33.246263 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:33.246521 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:33.249329 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "b324aeca0bef4bbea21e99e3591c03dd"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "120c6916accbf7623dbec7796d66c2ab"
server_key_iv: "e79369329f0ecbb15c4e01693c3e07b4"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.250048 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "b324aeca0bef4bbea21e99e3591c03dd"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "120c6916accbf7623dbec7796d66c2ab"
server_key_iv: "e79369329f0ecbb15c4e01693c3e07b4"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.255123 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:33.258474 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.259721 6682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:33.259866 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "b324aeca0bef4bbea21e99e3591c03dd"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "120c6916accbf7623dbec7796d66c2ab"
server_key_iv: "e79369329f0ecbb15c4e01693c3e07b4"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.260051 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:33.280913 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:33.285264 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:33.285563 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:33.286679 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:33.288261 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:33.288353 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.288487 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:33.288540 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.318697 6682 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:33731
I20260929 03:01:33.318763 6809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:33731 every 8 connection(s)
I20260929 03:01:33.319983 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:33.323968 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6682
I20260929 03:01:33.324146 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:33.324601 2870 external_mini_cluster.cc:1428] Setting key 3826433c86e1dd481794ed53474ce881
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:33.341118 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.322870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:39393 (local address 127.2.205.190:36501)
0929 03:01:33.323090 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:33.323097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:33.324470 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 03:01:33.326552 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:33.326557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:33.326640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:33.326721 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:33.328318 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.329077 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.330743 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.331193 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.331330 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.335142 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:33.335148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:33.335152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:33.335192 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:33.337862 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.338691 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.338695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.338698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.338844 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.339304 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.339315 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.339317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.339602 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:33.339740 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:33.340502 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 03:01:33.340776 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:01:33.342254 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.323479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36501 (local address 127.2.205.130:39393)
0929 03:01:33.324286 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:33.324334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:33.326256 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:33.326993 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:33.327001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:33.327535 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:33.328140 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:33.328159 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.329227 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.329231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:33.330586 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:33.330599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.331433 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.331440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:33.331697 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.332465 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:33.332490 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:33.334867 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:33.338049 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:33.338058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:33.338075 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:33.338551 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:33.338961 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:33.338965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:33.338969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:33.339190 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:33.339739 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:33.339749 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:33.340130 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 03:01:33.340390 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 03:01:33.341034 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:33.344007 6810 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36501
I20260929 03:01:33.344347 6810 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:33.345125 6810 heartbeater.cc:511] Master 127.2.205.190:36501 requested a full tablet report, sending...
I20260929 03:01:33.346704 6487 ts_manager.cc:194] Registered new tserver with Master: b324aeca0bef4bbea21e99e3591c03dd (127.2.205.130:33731)
I20260929 03:01:33.347741 6487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:39393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:33.366211 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:33.688326 6815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:33.688627 6815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:33.688784 6815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:33.696237 6815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:33.696367 6815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:33.696519 6815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:33.706858 6815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:33.708201 6815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:33.709542 6815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:33.722395 6820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:33.722456 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:33.722427 6821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:33.723177 6815 server_base.cc:1034] running on GCE node
I20260929 03:01:33.723817 6815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:33.724866 6815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:33.726074 6815 hybrid_clock.cc:648] HybridClock initialized: now 1790650893726018 us; error 51 us; skew 500 ppm
I20260929 03:01:33.726408 6815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:33.734174 6815 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:33.735795 6815 webserver.cc:460] Webserver started at http://127.2.205.131:45821/ using document root <none> and password file <none>
I20260929 03:01:33.736523 6815 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:33.736631 6815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:33.736964 6815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:33.740042 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "7ecb7d378248448486a24a75b9ed7336"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "680c5e2a115f02096643b357c45d4a66"
server_key_iv: "e7a8f9a121b65fb683631e29ce581fcd"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.740880 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "7ecb7d378248448486a24a75b9ed7336"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "680c5e2a115f02096643b357c45d4a66"
server_key_iv: "e7a8f9a121b65fb683631e29ce581fcd"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.746311 6815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 03:01:33.750090 6830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.751832 6815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:01:33.751988 6815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "7ecb7d378248448486a24a75b9ed7336"
format_stamp: "Formatted at 2026-09-29 03:01:33 on dist-test-slave-530n"
server_key: "680c5e2a115f02096643b357c45d4a66"
server_key_iv: "e7a8f9a121b65fb683631e29ce581fcd"
server_key_version: "encryptionkey@0"
I20260929 03:01:33.752164 6815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:33.764979 6815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:33.768832 6815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:33.769096 6815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:33.770031 6815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:33.771291 6815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:33.771361 6815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.771453 6815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:33.771508 6815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:33.801360 6815 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:35609
I20260929 03:01:33.801445 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:35609 every 8 connection(s)
I20260929 03:01:33.802702 6815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:33.804491 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6815
I20260929 03:01:33.804689 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:33.805195 2870 external_mini_cluster.cc:1428] Setting key 422674003b7528234c69997dee77604c
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:33.823057 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.805499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:43707 (local address 127.2.205.190:36501)
0929 03:01:33.805711 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:33.805717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:33.807179 (+ 1462us) server_negotiation.cc:407] Connection header received
0929 03:01:33.809108 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:33.809112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:33.809178 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:33.809258 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:33.810746 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.811497 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.813103 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.813504 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.813601 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.817052 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:33.817057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:33.817061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:33.817097 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:33.819796 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.820696 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.820700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.820703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.820887 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.821320 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.821324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.821326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.821677 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:33.821808 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:33.822586 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 03:01:33.822774 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:01:33.824095 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.806160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36501 (local address 127.2.205.131:43707)
0929 03:01:33.807027 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:33.807075 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:33.808824 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:33.809445 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:33.809454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:33.809984 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:33.810577 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:33.810596 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.811627 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.811631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:33.812963 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:33.812976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.813675 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.813680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:33.813899 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.814516 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:33.814538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:33.816836 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:33.819947 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:33.819959 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:33.819979 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:33.820573 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:33.820980 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:33.820983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:33.820986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:33.821182 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:33.821794 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:33.821800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:33.822168 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 03:01:33.822460 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 03:01:33.822894 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":349,"threads_started":1}
I20260929 03:01:33.825578 6943 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36501
I20260929 03:01:33.825904 6943 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:33.826699 6943 heartbeater.cc:511] Master 127.2.205.190:36501 requested a full tablet report, sending...
I20260929 03:01:33.828408 6487 ts_manager.cc:194] Registered new tserver with Master: 7ecb7d378248448486a24a75b9ed7336 (127.2.205.131:35609)
I20260929 03:01:33.829269 6487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:43707
I20260929 03:01:33.833137 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 03:01:33.861881 6676 heartbeater.cc:503] Master 127.2.205.190:36501 was elected leader, sending a full tablet report...
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:33.877435 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.863228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.2.205.129:46085)
0929 03:01:33.863935 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:33.863942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:33.863989 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:01:33.864071 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:33.864078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:33.864278 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:33.864426 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:33.865769 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.866785 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.867997 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.868544 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.868636 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.872317 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:33.872327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:33.872359 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:33.872390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:33.874913 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.875489 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.875496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.875498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.875616 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.875986 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.875989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.875991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.876318 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:33.876548 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:33.876923 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 03:01:33.877125 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":426,"threads_started":1}
Sep 29 03:01:33 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:33.896267 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.881638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.2.205.190:36501)
0929 03:01:33.881845 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:33.881851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:33.882100 (+ 249us) server_negotiation.cc:407] Connection header received
0929 03:01:33.882306 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:33.882310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:33.882379 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:33.882464 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:33.883748 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.884495 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.885848 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.886282 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.886401 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.890526 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:33.890532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:33.890536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:33.890581 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:33.893121 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.894059 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.894065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.894069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.894236 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.894665 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.894671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.894675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.895152 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:33.895404 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:33.895721 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 03:01:33.895943 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:01:33.896919 6487 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116
I20260929 03:01:33.897959 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6548
I20260929 03:01:33.910189 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:33.898818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.2.205.129:46085)
0929 03:01:33.898974 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:33.898979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:33.899193 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:01:33.899334 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:33.899338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:33.899447 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:33.899525 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:33.900643 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.901365 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.902525 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:33.902936 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:33.903024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:33.905192 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:33.905197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:33.905201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:33.905234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:33.907729 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.908414 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.908419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.908421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.908573 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:33.909056 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:33.909060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:33.909062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:33.909331 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:33.909546 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:33.909786 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:01:33.909953 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:01:33.910837 6671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:33.995342 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6548
I20260929 03:01:34.008630 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6682
Sep 29 03:01:34 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:34.024540 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:34.009705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.2.205.130:33731)
0929 03:01:34.010303 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:34.010310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:34.010337 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:01:34.010407 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:34.010414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:34.010645 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:34.010787 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:34.011977 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:34.013076 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:34.014389 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:34.014973 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:34.015084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:34.019187 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:34.019202 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:34.019217 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:34.019259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:34.021910 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:34.022628 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:34.022637 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:34.022639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:34.022762 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:34.023141 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:34.023144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:34.023146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:34.023533 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:34.023765 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:34.024021 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 03:01:34.024224 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
I20260929 03:01:34.025895 6805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:34.106987 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6682
I20260929 03:01:34.120954 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6815
Sep 29 03:01:34 dist-test-slave-530n krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:34.137377 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:34.122069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.2.205.131:35609)
0929 03:01:34.122777 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:34.122785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:34.122818 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:01:34.122890 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:34.122897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:34.123154 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:34.123327 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:34.124782 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:34.125843 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:34.127236 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:34.127941 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:34.128086 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:34.132324 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:34.132337 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:34.132353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:34.132389 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:34.134979 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:34.135586 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:34.135594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:34.135596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:34.135717 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:34.135969 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:34.135972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:34.135974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:34.136307 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:34.136543 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:34.136889 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 03:01:34.137096 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":338,"threads_started":1}
I20260929 03:01:34.138645 6938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:34.221283 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6815
I20260929 03:01:34.236689 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6455
I20260929 03:01:34.237983 6517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:34.340133 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6455
2026-09-29T03:01:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3718 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:34 dist-test-slave-530n krb5kdc[6962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:34 dist-test-slave-530n krb5kdc[6962](info): set up 2 sockets
Sep 29 03:01:34 dist-test-slave-530n krb5kdc[6962](info): commencing operation
krb5kdc: starting...
W20260929 03:01:35.361202 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:35 dist-test-slave-530n krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:35.549082 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36731
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:35.867470 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:35.867762 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:35.867889 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:35.875097 6975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:35.875198 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:35.875265 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:35.875336 6975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:35.875396 6975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:35.885810 6975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:35.887238 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:35.888648 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:35.900588 6981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:35.900599 6983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:35.901165 6980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:35.901309 6975 server_base.cc:1034] running on GCE node
I20260929 03:01:35.902161 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:35.903643 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:35.904846 6975 hybrid_clock.cc:648] HybridClock initialized: now 1790650895904783 us; error 51 us; skew 500 ppm
I20260929 03:01:35.905172 6975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:35 dist-test-slave-530n krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:35.908203 6975 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:35.909412 6975 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:35.909515 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:35.909884 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:35.913149 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "a49bd81e630c40ac9f3e9fface904af0"
format_stamp: "Formatted at 2026-09-29 03:01:35 on dist-test-slave-530n"
server_key: "d24e0268da6a54ad6a92ad2f391941fa"
server_key_iv: "432d07f80c9437578cdf70b5ae9bd077"
server_key_version: "encryptionkey@0"
I20260929 03:01:35.914031 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "a49bd81e630c40ac9f3e9fface904af0"
format_stamp: "Formatted at 2026-09-29 03:01:35 on dist-test-slave-530n"
server_key: "d24e0268da6a54ad6a92ad2f391941fa"
server_key_iv: "432d07f80c9437578cdf70b5ae9bd077"
server_key_version: "encryptionkey@0"
I20260929 03:01:35.919687 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 03:01:35.923664 6989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:35.925171 6975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:35.925364 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "a49bd81e630c40ac9f3e9fface904af0"
format_stamp: "Formatted at 2026-09-29 03:01:35 on dist-test-slave-530n"
server_key: "d24e0268da6a54ad6a92ad2f391941fa"
server_key_iv: "432d07f80c9437578cdf70b5ae9bd077"
server_key_version: "encryptionkey@0"
I20260929 03:01:35.925639 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:35.939365 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:35.943678 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:35.943908 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:35.960711 6975 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36731
I20260929 03:01:35.960690 7040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36731 every 8 connection(s)
I20260929 03:01:35.961582 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:35.965097 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:35.966626 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 6975
I20260929 03:01:35.966828 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:35.967401 2870 external_mini_cluster.cc:1428] Setting key f8642842f0407e8740b8870513336bd0
I20260929 03:01:35.976308 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: Bootstrap starting.
I20260929 03:01:35.980109 7041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:35.981421 7041 log.cc:824] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:01:35 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:35.984297 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: No bootstrap required, opened a new log
I20260929 03:01:35.990195 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:35.969311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.2.205.190:36731)
0929 03:01:35.970086 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:35.970098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:35.970143 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:01:35.971737 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:35.971758 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:35.972141 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:35.972450 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:35.974235 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:35.975470 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:35.977234 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:35.977899 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:35.978038 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:35.982706 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:35.982715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:35.982731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:35.982770 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:35.985756 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:35.986535 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:35.986542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:35.986549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:35.986743 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:35.987146 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:35.987150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:35.987152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:35.987780 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:35.988072 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:35.988582 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 03:01:35.989078 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
I20260929 03:01:35.992071 7041 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } }
I20260929 03:01:35.992457 7041 raft_consensus.cc:374] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:35.992564 7041 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49bd81e630c40ac9f3e9fface904af0, State: Initialized, Role: FOLLOWER
I20260929 03:01:35.993942 7041 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } }
I20260929 03:01:35.994184 7041 raft_consensus.cc:388] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:35.994311 7041 raft_consensus.cc:482] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:35.994470 7041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:35.996932 7041 raft_consensus.cc:504] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } }
I20260929 03:01:35.997475 7041 leader_election.cc:302] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49bd81e630c40ac9f3e9fface904af0; no voters:
I20260929 03:01:35.998078 7041 leader_election.cc:288] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:35.998191 7046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:35.998911 7046 raft_consensus.cc:686] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [term 1 LEADER]: Becoming Leader. State: Replica: a49bd81e630c40ac9f3e9fface904af0, State: Running, Role: LEADER
I20260929 03:01:35.999461 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } }
I20260929 03:01:36.000500 7041 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:36.002624 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a49bd81e630c40ac9f3e9fface904af0. Latest consensus state: current_term: 1 leader_uuid: "a49bd81e630c40ac9f3e9fface904af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } } }
I20260929 03:01:36.003268 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:36.002619 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a49bd81e630c40ac9f3e9fface904af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49bd81e630c40ac9f3e9fface904af0" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36731 } } }
I20260929 03:01:36.004415 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:36.009848 7053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:36.014312 7053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:36.022552 7053 catalog_manager.cc:1347] Generated new cluster ID: 19fed949dac04d8e8d0c35e4ba5f4d53
I20260929 03:01:36.022683 7053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:36.030063 7053 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:36.031414 7053 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:36.042559 7053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: Generated new TSK 0
I20260929 03:01:36.043725 7053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:36.084534 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:36.409902 7067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:36.410213 7067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:36.410341 7067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:36.417785 7067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:36.417915 7067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:36.418075 7067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:36.428182 7067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:36.429579 7067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:36.430903 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:36.443138 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:36.443259 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:36.443256 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:36.444203 7067 server_base.cc:1034] running on GCE node
I20260929 03:01:36.444868 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:36.445923 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:36.447186 7067 hybrid_clock.cc:648] HybridClock initialized: now 1790650896447122 us; error 55 us; skew 500 ppm
I20260929 03:01:36.447549 7067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:36 dist-test-slave-530n krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:36.450832 7067 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:36.452087 7067 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:36.452203 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:36.452610 7067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:36.455849 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "cba07ce6be6c498484dcaa8be63eff11"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "e8b7b64324ed1ae8a019a4303453e7b1"
server_key_iv: "cd132553865e8772c786c832f8a1bc58"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.456676 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "cba07ce6be6c498484dcaa8be63eff11"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "e8b7b64324ed1ae8a019a4303453e7b1"
server_key_iv: "cd132553865e8772c786c832f8a1bc58"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.461824 7067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:36.465281 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:36.466637 7067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:01:36.466799 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "cba07ce6be6c498484dcaa8be63eff11"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "e8b7b64324ed1ae8a019a4303453e7b1"
server_key_iv: "cd132553865e8772c786c832f8a1bc58"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.467007 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:36.486157 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:36.490844 7067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:36.491106 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:36.492172 7067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:36.493716 7067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:36.493831 7067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:36.493942 7067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:36.494009 7067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:36.523995 7067 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37947
I20260929 03:01:36.524047 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37947 every 8 connection(s)
I20260929 03:01:36.524863 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:36.532994 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7067
I20260929 03:01:36.533214 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:36.533912 2870 external_mini_cluster.cc:1428] Setting key c29d9c690ec730c28a338e1a1e79cd9b
Sep 29 03:01:36 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:36.546432 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:36.527198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45715 (local address 127.2.205.190:36731)
0929 03:01:36.527688 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:36.527695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:36.528602 (+ 907us) server_negotiation.cc:407] Connection header received
0929 03:01:36.530497 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:36.530503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:36.530584 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:36.530709 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:36.532433 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:36.533522 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:36.535299 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:36.535788 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:36.535923 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:36.539637 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:36.539644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:36.539648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:36.539688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:36.542960 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:36.543931 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:36.543935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:36.543938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:36.544070 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:36.544520 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:36.544522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:36.544524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:36.544886 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:36.545013 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:36.545949 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 03:01:36.546151 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260929 03:01:36.547878 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:36.527784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36731 (local address 127.2.205.129:45715)
0929 03:01:36.528448 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:36.528486 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:36.530203 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:36.530959 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:36.530973 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:36.531688 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:36.532270 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:36.532292 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:36.533662 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:36.533666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:36.535150 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:36.535164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:36.535994 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:36.535999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:36.536210 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:36.536968 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:36.536994 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:36.539355 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:36.543131 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:36.543141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:36.543161 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:36.543813 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:36.544169 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:36.544173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:36.544177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:36.544427 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:36.544999 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:36.545007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:36.545517 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 03:01:36.545834 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 03:01:36.546429 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":252,"threads_started":1}
I20260929 03:01:36.549718 7194 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36731
I20260929 03:01:36.550071 7194 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:36.550918 7194 heartbeater.cc:511] Master 127.2.205.190:36731 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:36.553341 7006 ts_manager.cc:194] Registered new tserver with Master: cba07ce6be6c498484dcaa8be63eff11 (127.2.205.129:37947)
I20260929 03:01:36.554970 7006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:45715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:36.576336 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:36.895656 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:36.895987 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:36.896133 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:36.903803 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:36.903946 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:36.904109 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:36.914261 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:36.915867 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:36.917478 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:36.930673 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:36.930784 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:36.931048 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:36.931697 7200 server_base.cc:1034] running on GCE node
I20260929 03:01:36.932374 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:36.933414 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:36.934646 7200 hybrid_clock.cc:648] HybridClock initialized: now 1790650896934588 us; error 47 us; skew 500 ppm
I20260929 03:01:36.934993 7200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:36 dist-test-slave-530n krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:36.938102 7200 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:36.939284 7200 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:36.939378 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:36.939656 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:36.942802 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "d0f9baa2789141bead0b81c81bee5cdd"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "2f27c2acfd713ee4f7caafd2c7b43617"
server_key_iv: "124ad6ecabefcc64090c24e1916b199f"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.943612 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "d0f9baa2789141bead0b81c81bee5cdd"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "2f27c2acfd713ee4f7caafd2c7b43617"
server_key_iv: "124ad6ecabefcc64090c24e1916b199f"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.948676 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 03:01:36.951921 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:36.953218 7200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 03:01:36.953361 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "d0f9baa2789141bead0b81c81bee5cdd"
format_stamp: "Formatted at 2026-09-29 03:01:36 on dist-test-slave-530n"
server_key: "2f27c2acfd713ee4f7caafd2c7b43617"
server_key_iv: "124ad6ecabefcc64090c24e1916b199f"
server_key_version: "encryptionkey@0"
I20260929 03:01:36.953548 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:36.983151 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:36.987468 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:36.987738 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:36.988736 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:36.990119 7200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:36.990204 7200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:36.990302 7200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:36.990371 7200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:37.020692 7200 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:43355
I20260929 03:01:37.020732 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:43355 every 8 connection(s)
I20260929 03:01:37.021662 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:37.025947 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7200
I20260929 03:01:37.026149 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:37.026696 2870 external_mini_cluster.cc:1428] Setting key 050de886d75b14cedde085f8ed9e1c3d
Sep 29 03:01:37 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:37.044421 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.024202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:60363 (local address 127.2.205.190:36731)
0929 03:01:37.024370 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.024375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.025586 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 03:01:37.027995 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.027999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.028064 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.028144 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:37.030226 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.031022 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.033144 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.033570 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.033664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.037540 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:37.037547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:37.037552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:37.037592 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:37.040852 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.041708 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.041714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.041717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.041882 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.042392 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.042397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.042400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.042792 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:37.042954 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.044001 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.044186 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:01:37.046022 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.024666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36731 (local address 127.2.205.130:60363)
0929 03:01:37.025415 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.025455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.027684 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.028424 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.028446 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.029183 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:37.030051 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.030078 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.031169 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.031174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.033010 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.033025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.033781 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.033787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.034063 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.034782 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:37.034806 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:37.037229 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:37.041023 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:37.041032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:37.041050 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:37.041569 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:37.042011 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:37.042015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:37.042017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:37.042237 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:37.042932 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:37.042940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:37.043534 (+ 594us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.043884 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.044519 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:37.047500 7327 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36731
I20260929 03:01:37.047847 7327 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:37.048525 7327 heartbeater.cc:511] Master 127.2.205.190:36731 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:37.050055 7006 ts_manager.cc:194] Registered new tserver with Master: d0f9baa2789141bead0b81c81bee5cdd (127.2.205.130:43355)
I20260929 03:01:37.050823 7006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:60363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:37.069813 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:37.391292 7332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:37.391610 7332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:37.391707 7332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:37.398911 7332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:37.399030 7332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:37.399147 7332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:37.409128 7332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:37.410518 7332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:37.411888 7332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:37.425458 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:37.425520 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:37.425501 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:37.426435 7332 server_base.cc:1034] running on GCE node
I20260929 03:01:37.427067 7332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:37.428046 7332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:37.429268 7332 hybrid_clock.cc:648] HybridClock initialized: now 1790650897429193 us; error 55 us; skew 500 ppm
I20260929 03:01:37.429625 7332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:37 dist-test-slave-530n krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:37.432647 7332 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:37.433768 7332 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:37.433857 7332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:37.434111 7332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:37.436991 7332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "38a1cb08e4034550afaf8d0aa08cbf65"
format_stamp: "Formatted at 2026-09-29 03:01:37 on dist-test-slave-530n"
server_key: "c97d7a4316aca99c27d1219a3b25ab57"
server_key_iv: "36ef7e6744e6977576213ef85baa498c"
server_key_version: "encryptionkey@0"
I20260929 03:01:37.437748 7332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "38a1cb08e4034550afaf8d0aa08cbf65"
format_stamp: "Formatted at 2026-09-29 03:01:37 on dist-test-slave-530n"
server_key: "c97d7a4316aca99c27d1219a3b25ab57"
server_key_iv: "36ef7e6744e6977576213ef85baa498c"
server_key_version: "encryptionkey@0"
I20260929 03:01:37.442664 7332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:37.446069 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:37.447417 7332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:37.447554 7332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "38a1cb08e4034550afaf8d0aa08cbf65"
format_stamp: "Formatted at 2026-09-29 03:01:37 on dist-test-slave-530n"
server_key: "c97d7a4316aca99c27d1219a3b25ab57"
server_key_iv: "36ef7e6744e6977576213ef85baa498c"
server_key_version: "encryptionkey@0"
I20260929 03:01:37.447739 7332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:37.466562 7332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:37.470877 7332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:37.471148 7332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:37.472164 7332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:37.473544 7332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:37.473634 7332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:37.473735 7332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:37.473798 7332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:37.504562 7332 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:38581
I20260929 03:01:37.504608 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:38581 every 8 connection(s)
I20260929 03:01:37.505412 7332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:37.507186 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7332
I20260929 03:01:37.507385 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:37.507900 2870 external_mini_cluster.cc:1428] Setting key e35750693c8683b60dfb0bb0110f817d
Sep 29 03:01:37 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650897, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:37.526043 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.507874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:50419 (local address 127.2.205.190:36731)
0929 03:01:37.508049 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.508054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.509525 (+ 1471us) server_negotiation.cc:407] Connection header received
0929 03:01:37.511706 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.511711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.511835 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.511919 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:37.513444 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.514197 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.515717 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.516137 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.516299 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.519710 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:37.519715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:37.519719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:37.519754 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:37.522345 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.523359 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.523363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.523366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.523494 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.524058 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.524063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.524067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.524474 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:37.524637 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.525443 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.525691 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:01:37.527027 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.508549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36731 (local address 127.2.205.131:50419)
0929 03:01:37.509365 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.509414 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.511450 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.512171 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.512179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.512680 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:37.513284 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.513301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.514346 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.514350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.515583 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.515595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.516390 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.516397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.516617 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.517301 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:37.517332 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:37.519472 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:37.522531 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:37.522540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:37.522564 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:37.523203 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:37.523638 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:37.523643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:37.523647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:37.523917 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:37.524640 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:37.524647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:37.525055 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.525301 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.525740 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":313,"threads_started":1}
I20260929 03:01:37.528447 7459 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36731
I20260929 03:01:37.528829 7459 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:37.529707 7459 heartbeater.cc:511] Master 127.2.205.190:36731 requested a full tablet report, sending...
I20260929 03:01:37.531237 7006 ts_manager.cc:194] Registered new tserver with Master: 38a1cb08e4034550afaf8d0aa08cbf65 (127.2.205.131:38581)
I20260929 03:01:37.532060 7006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:50419
I20260929 03:01:37.536437 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:01:37.553311 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.541020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.2.205.190:36731)
0929 03:01:37.541192 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.541197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.541653 (+ 456us) server_negotiation.cc:407] Connection header received
0929 03:01:37.541798 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.541801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.541880 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.541969 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:37.543550 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.544278 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.545625 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.546065 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.546168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.548338 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:37.548343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:37.548346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:37.548381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:37.550705 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.551358 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.551363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.551365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.551495 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:37.551844 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:37.551850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:37.551853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:37.552306 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:37.552498 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.552840 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.553009 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:37.558349 7194 heartbeater.cc:503] Master 127.2.205.190:36731 was elected leader, sending a full tablet report...
I20260929 03:01:37.557480 7006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:37.564604 7006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:37.594321 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.577157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.2.205.131:38581)
0929 03:01:37.578090 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.578099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.578491 (+ 392us) server_negotiation.cc:407] Connection header received
0929 03:01:37.578739 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.578747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.579027 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.586282 (+ 7255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:37.586674 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.588556 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.591412 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.592868 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.593010 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.593253 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.593778 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.593961 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":571,"threads_started":1}
I20260929 03:01:37.595459 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.576935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:38581 (local address 127.0.0.1:46776)
0929 03:01:37.578354 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.578389 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.578549 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.585289 (+ 6740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.585299 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.585344 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:37.585961 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.585992 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.588796 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.588803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.591230 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.591242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.593076 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.593083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.593312 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.593393 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.593546 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.595040 (+ 1494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":1258,"thread_start_us":1082,"threads_started":1}
I20260929 03:01:37.596721 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.579280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37947 (local address 127.0.0.1:47652)
0929 03:01:37.589265 (+ 9985us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.589285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.589418 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.590093 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.590098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.590121 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:37.590594 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.590607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.592239 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.592244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.594110 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.594122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.595923 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.595930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.596087 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.596117 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.596200 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.596337 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9852,"thread_start_us":11024,"threads_started":1}
I20260929 03:01:37.596769 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.579616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.2.205.129:37947)
0929 03:01:37.587239 (+ 7623us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.587246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.589557 (+ 2311us) server_negotiation.cc:407] Connection header received
0929 03:01:37.589662 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.589669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.589907 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.590091 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:37.590784 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.592118 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.594294 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.595713 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.595857 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.596090 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.596255 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.596422 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7454,"thread_start_us":7286,"threads_started":1}
I20260929 03:01:37.597553 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.578733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:43355 (local address 127.0.0.1:40724)
0929 03:01:37.579272 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.579292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.579411 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.591475 (+ 12064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.591479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.591508 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:37.591822 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.591833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.593262 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.593267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.595111 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.595123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.596950 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.596956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.597120 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.597150 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.597241 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.597343 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260929 03:01:37.597795 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.578999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.2.205.130:43355)
0929 03:01:37.590894 (+ 11895us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.590903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.590931 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:01:37.591024 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.591032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.591286 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.591482 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:37.591991 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.593102 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.595266 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.596487 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.596612 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.596816 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.597307 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.597494 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11715,"thread_start_us":11612,"threads_started":1}
I20260929 03:01:37.599711 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 (DEFAULT_TABLE table=test-table [id=d3c04db3d72448c995f8d4f84c7f8015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:37.601084 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 (DEFAULT_TABLE table=test-table [id=d3c04db3d72448c995f8d4f84c7f8015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:37.601590 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1d9de59eca4a3db03d240066a29e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:37.602324 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1d9de59eca4a3db03d240066a29e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:37.602764 7262 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 (DEFAULT_TABLE table=test-table [id=d3c04db3d72448c995f8d4f84c7f8015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:37.604198 7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1d9de59eca4a3db03d240066a29e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:37.613643 7477 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Bootstrap starting.
I20260929 03:01:37.616852 7478 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Bootstrap starting.
I20260929 03:01:37.618739 7477 tablet_bootstrap.cc:654] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:37.619937 7479 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Bootstrap starting.
I20260929 03:01:37.620253 7477 log.cc:824] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:37.620712 7478 tablet_bootstrap.cc:654] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:37.622123 7478 log.cc:824] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:37.622941 7479 tablet_bootstrap.cc:654] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:37.624256 7479 log.cc:824] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:37.625391 7478 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: No bootstrap required, opened a new log
I20260929 03:01:37.625483 7477 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: No bootstrap required, opened a new log
I20260929 03:01:37.625676 7478 ts_tablet_manager.cc:1397] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 03:01:37.625804 7477 ts_tablet_manager.cc:1397] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260929 03:01:37.627424 7479 tablet_bootstrap.cc:492] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: No bootstrap required, opened a new log
I20260929 03:01:37.627810 7479 ts_tablet_manager.cc:1397] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 03:01:37.633675 7478 raft_consensus.cc:348] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.634092 7478 raft_consensus.cc:374] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:37.634209 7478 raft_consensus.cc:729] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a1cb08e4034550afaf8d0aa08cbf65, State: Initialized, Role: FOLLOWER
I20260929 03:01:37.634856 7478 consensus_queue.cc:260] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.635982 7459 heartbeater.cc:503] Master 127.2.205.190:36731 was elected leader, sending a full tablet report...
I20260929 03:01:37.636368 7477 raft_consensus.cc:348] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.636734 7478 ts_tablet_manager.cc:1428] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 03:01:37.636838 7477 raft_consensus.cc:374] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:37.636989 7477 raft_consensus.cc:729] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba07ce6be6c498484dcaa8be63eff11, State: Initialized, Role: FOLLOWER
I20260929 03:01:37.637683 7477 consensus_queue.cc:260] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.638367 7479 raft_consensus.cc:348] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.638902 7479 raft_consensus.cc:374] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:37.639006 7479 raft_consensus.cc:729] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f9baa2789141bead0b81c81bee5cdd, State: Initialized, Role: FOLLOWER
I20260929 03:01:37.639568 7477 ts_tablet_manager.cc:1428] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 03:01:37.639645 7479 consensus_queue.cc:260] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.640771 7327 heartbeater.cc:503] Master 127.2.205.190:36731 was elected leader, sending a full tablet report...
I20260929 03:01:37.641449 7479 ts_tablet_manager.cc:1428] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 03:01:37.756842 7460 tablet.cc:2369] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:01:37.773449 7328 tablet.cc:2369] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:01:37.776932 7195 tablet.cc:2369] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:37.886370 7484 raft_consensus.cc:482] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:37.886693 7484 raft_consensus.cc:504] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.888504 7484 leader_election.cc:288] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a1cb08e4034550afaf8d0aa08cbf65 (127.2.205.131:38581), d0f9baa2789141bead0b81c81bee5cdd (127.2.205.130:43355)
I20260929 03:01:37.897053 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.889035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:34365 (local address 127.2.205.131:38581)
0929 03:01:37.889259 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.889267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.889798 (+ 531us) server_negotiation.cc:407] Connection header received
0929 03:01:37.890029 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.890036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.890126 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.890233 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:37.890964 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.892024 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.894689 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.895918 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.896080 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.896162 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.896608 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.896780 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:01:37.897053 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.890025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45263 (local address 127.2.205.130:43355)
0929 03:01:37.890214 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.890220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.890687 (+ 467us) server_negotiation.cc:407] Connection header received
0929 03:01:37.890857 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.890861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.890932 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.891026 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:37.891483 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.892374 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.894698 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.895919 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.896080 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.896162 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.896607 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.896796 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:37.897315 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.889843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:43355 (local address 127.2.205.129:45263)
0929 03:01:37.890576 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.890598 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.890721 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.891046 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.891049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.891070 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:37.891334 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.891342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.892503 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.892507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.894536 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.894545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.896210 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.896216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.896419 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.896468 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.896571 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.896780 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":64,"thread_start_us":383,"threads_started":1}
I20260929 03:01:37.897280 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.888957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:38581 (local address 127.2.205.129:34365)
0929 03:01:37.889655 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:37.889676 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:37.889843 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:37.890328 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:37.890333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:37.890366 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:37.890818 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.890831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.892178 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.892182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.894513 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:37.894528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.896183 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.896193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:37.896480 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.896506 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:01:37.896577 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:01:37.896767 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":101,"thread_start_us":407,"threads_started":1}
I20260929 03:01:37.898546 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1d9de59eca4a3db03d240066a29e53" candidate_uuid: "cba07ce6be6c498484dcaa8be63eff11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f9baa2789141bead0b81c81bee5cdd" is_pre_election: true
I20260929 03:01:37.898546 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1d9de59eca4a3db03d240066a29e53" candidate_uuid: "cba07ce6be6c498484dcaa8be63eff11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" is_pre_election: true
I20260929 03:01:37.899148 7414 raft_consensus.cc:2454] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cba07ce6be6c498484dcaa8be63eff11 in term 0.
I20260929 03:01:37.899148 7282 raft_consensus.cc:2454] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cba07ce6be6c498484dcaa8be63eff11 in term 0.
I20260929 03:01:37.899854 7082 leader_election.cc:302] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a1cb08e4034550afaf8d0aa08cbf65, cba07ce6be6c498484dcaa8be63eff11; no voters:
I20260929 03:01:37.900230 7484 raft_consensus.cc:2781] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:37.900400 7484 raft_consensus.cc:482] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:37.900501 7484 raft_consensus.cc:3037] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:37.902773 7484 raft_consensus.cc:504] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.903357 7484 leader_election.cc:288] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [CANDIDATE]: Term 1 election: Requested vote from peers 38a1cb08e4034550afaf8d0aa08cbf65 (127.2.205.131:38581), d0f9baa2789141bead0b81c81bee5cdd (127.2.205.130:43355)
I20260929 03:01:37.903920 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1d9de59eca4a3db03d240066a29e53" candidate_uuid: "cba07ce6be6c498484dcaa8be63eff11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f9baa2789141bead0b81c81bee5cdd"
I20260929 03:01:37.903918 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1d9de59eca4a3db03d240066a29e53" candidate_uuid: "cba07ce6be6c498484dcaa8be63eff11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a1cb08e4034550afaf8d0aa08cbf65"
I20260929 03:01:37.904142 7414 raft_consensus.cc:3037] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:37.904145 7282 raft_consensus.cc:3037] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:37.907305 7414 raft_consensus.cc:2454] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cba07ce6be6c498484dcaa8be63eff11 in term 1.
I20260929 03:01:37.907305 7282 raft_consensus.cc:2454] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cba07ce6be6c498484dcaa8be63eff11 in term 1.
I20260929 03:01:37.907706 7082 leader_election.cc:302] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a1cb08e4034550afaf8d0aa08cbf65, cba07ce6be6c498484dcaa8be63eff11; no voters:
I20260929 03:01:37.907991 7484 raft_consensus.cc:2781] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:37.908432 7484 raft_consensus.cc:686] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 1 LEADER]: Becoming Leader. State: Replica: cba07ce6be6c498484dcaa8be63eff11, State: Running, Role: LEADER
I20260929 03:01:37.908941 7484 consensus_queue.cc:237] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } }
I20260929 03:01:37.913322 7004 catalog_manager.cc:5495] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 reported cstate change: term changed from 0 to 1, leader changed from <none> to cba07ce6be6c498484dcaa8be63eff11 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "cba07ce6be6c498484dcaa8be63eff11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cba07ce6be6c498484dcaa8be63eff11" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37947 } health_report { overall_health: HEALTHY } } }
I20260929 03:01:37.956295 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:37.951066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.2.205.129:37947)
0929 03:01:37.951249 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:37.951256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:37.951358 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:01:37.951560 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:37.951564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:37.951649 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:37.951787 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:37.952122 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.952843 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.954322 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:37.954710 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:37.954824 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:37.955260 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:37.955742 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:37.955875 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:37.955974 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:01:37.956051 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:01:37.964200 7282 raft_consensus.cc:1261] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 1 FOLLOWER]: Refusing update from remote peer cba07ce6be6c498484dcaa8be63eff11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:37.964385 7414 raft_consensus.cc:1261] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 1 FOLLOWER]: Refusing update from remote peer cba07ce6be6c498484dcaa8be63eff11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:37.965176 7488 consensus_queue.cc:1035] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f9baa2789141bead0b81c81bee5cdd" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:37.965516 7484 consensus_queue.cc:1035] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a1cb08e4034550afaf8d0aa08cbf65" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:37.979147 7491 mvcc.cc:205] Tried to move back new op lower bound from 7334506078051942400 to 7334506077838594048. Current Snapshot: MvccSnapshot[applied={T|T < 7334506078051942400}]
I20260929 03:01:37.981518 7492 mvcc.cc:205] Tried to move back new op lower bound from 7334506078051942400 to 7334506077838594048. Current Snapshot: MvccSnapshot[applied={T|T < 7334506078051942400}]
I20260929 03:01:37.995071 7006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:37.995412 7006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:37.999100 7006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49bd81e630c40ac9f3e9fface904af0: Sending DeleteTablet for 3 replicas of tablet 9e1d9de59eca4a3db03d240066a29e53
I20260929 03:01:38.000197 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:37 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46776
I20260929 03:01:38.000401 7262 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:37 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:40724
I20260929 03:01:38.000779 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1d9de59eca4a3db03d240066a29e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:37 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:47652
I20260929 03:01:38.004042 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7067
I20260929 03:01:38.029863 7504 tablet_replica.cc:330] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: stopping tablet replica
I20260929 03:01:38.031750 7504 raft_consensus.cc:2229] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:38.032662 7504 raft_consensus.cc:2258] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:38.038754 7504 ts_tablet_manager.cc:1905] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:38.039407 7502 tablet_replica.cc:330] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: stopping tablet replica
I20260929 03:01:38.040021 7503 tablet_replica.cc:330] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: stopping tablet replica
I20260929 03:01:38.041098 7502 raft_consensus.cc:2229] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:38.041227 7503 raft_consensus.cc:2229] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:38.041764 7502 raft_consensus.cc:2258] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:38.041762 7503 raft_consensus.cc:2258] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:38.046137 7502 ts_tablet_manager.cc:1905] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:38.049275 7504 ts_tablet_manager.cc:1918] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 03:01:38 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:38.049535 7504 log.cc:1196] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/9e1d9de59eca4a3db03d240066a29e53
I20260929 03:01:38.050122 7504 ts_tablet_manager.cc:1939] T 9e1d9de59eca4a3db03d240066a29e53 P cba07ce6be6c498484dcaa8be63eff11: Deleting consensus metadata
I20260929 03:01:38.051671 6990 catalog_manager.cc:4841] TS cba07ce6be6c498484dcaa8be63eff11 (127.2.205.129:37947): tablet 9e1d9de59eca4a3db03d240066a29e53 (table test-table [id=d3c04db3d72448c995f8d4f84c7f8015]) successfully deleted
I20260929 03:01:38.053118 7503 ts_tablet_manager.cc:1905] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:38.055483 7502 ts_tablet_manager.cc:1918] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:38.055716 7502 log.cc:1196] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/9e1d9de59eca4a3db03d240066a29e53
I20260929 03:01:38.056241 7502 ts_tablet_manager.cc:1939] T 9e1d9de59eca4a3db03d240066a29e53 P 38a1cb08e4034550afaf8d0aa08cbf65: Deleting consensus metadata
I20260929 03:01:38.058140 6990 catalog_manager.cc:4841] TS 38a1cb08e4034550afaf8d0aa08cbf65 (127.2.205.131:38581): tablet 9e1d9de59eca4a3db03d240066a29e53 (table test-table [id=d3c04db3d72448c995f8d4f84c7f8015]) successfully deleted
I20260929 03:01:38.059684 7503 ts_tablet_manager.cc:1918] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:38.059856 7503 log.cc:1196] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/9e1d9de59eca4a3db03d240066a29e53
I20260929 03:01:38.060364 7503 ts_tablet_manager.cc:1939] T 9e1d9de59eca4a3db03d240066a29e53 P d0f9baa2789141bead0b81c81bee5cdd: Deleting consensus metadata
I20260929 03:01:38.061738 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:38.005452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.2.205.129:37947)
0929 03:01:38.005630 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:38.005636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:38.027660 (+ 22024us) server_negotiation.cc:407] Connection header received
0929 03:01:38.028220 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:38.028228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:38.028330 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:38.028410 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:38.032999 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.034256 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.042310 (+ 8054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.043042 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.043188 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:38.054161 (+ 10973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:38.054175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:38.054192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:38.054237 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:38.058226 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.059134 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.059143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.059148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.059328 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.059779 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.059784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.059787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.060231 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:38.060525 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:38.061107 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 03:01:38.061361 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:38.062218 6990 catalog_manager.cc:4841] TS d0f9baa2789141bead0b81c81bee5cdd (127.2.205.130:43355): tablet 9e1d9de59eca4a3db03d240066a29e53 (table test-table [id=d3c04db3d72448c995f8d4f84c7f8015]) successfully deleted
I20260929 03:01:38.063007 7189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:38.164228 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7067
I20260929 03:01:38.179414 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7200
Sep 29 03:01:38 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:38.194348 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:38.180128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.2.205.130:43355)
0929 03:01:38.180310 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:38.180315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:38.180495 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:01:38.180638 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:38.180642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:38.180717 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:38.180841 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:38.182033 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.182795 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.184030 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.184441 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.184540 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:38.188863 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:38.188881 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:38.188897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:38.188941 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:38.191726 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.192403 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.192411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.192413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.192562 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.193043 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.193047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.193049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.193439 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:38.193650 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:38.193890 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:01:38.194058 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:01:38.194990 7322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:38.283075 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7200
I20260929 03:01:38.296622 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7332
Sep 29 03:01:38 dist-test-slave-530n krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:38.311623 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:38.297496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.2.205.131:38581)
0929 03:01:38.297704 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:38.297709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:38.297730 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:01:38.297891 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:38.297895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:38.297962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:38.298045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:38.299299 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.300034 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.301377 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:38.301824 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:38.301936 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:38.306173 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:38.306189 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:38.306205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:38.306247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:38.308951 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.309598 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.309606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.309608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.309747 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:38.310111 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:38.310115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:38.310117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:38.310501 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:38.310780 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:38.311078 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 03:01:38.311294 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:01:38.312268 7454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:38.399273 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7332
I20260929 03:01:38.412386 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 6975
I20260929 03:01:38.413556 7036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:38.521838 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 6975
2026-09-29T03:01:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4182 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:38Z Disabled control of system clock
I20260929 03:01:38.581950 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39025
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:01:38.902120 7513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:38.902472 7513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:38.902555 7513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:38.909878 7513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:38.909999 7513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:38.910037 7513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:38.910096 7513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:38.910140 7513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:38.919472 7513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:38.920858 7513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:38.922725 7513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:38.934351 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:38.934294 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:38.934386 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:38.935029 7513 server_base.cc:1034] running on GCE node
I20260929 03:01:38.935748 7513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:38.937104 7513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:38.938320 7513 hybrid_clock.cc:648] HybridClock initialized: now 1790650898938245 us; error 46 us; skew 500 ppm
I20260929 03:01:38.938650 7513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:38.946228 7513 webserver.cc:460] Webserver started at http://127.2.205.190:41101/ using document root <none> and password file <none>
I20260929 03:01:38.946904 7513 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:38.946995 7513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:38.947294 7513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:38.950112 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "b3e08ff429874e0c969fb26d16786b94"
format_stamp: "Formatted at 2026-09-29 03:01:38 on dist-test-slave-530n"
server_key: "98884381badc41cabfc3988161bbd5dc"
server_key_iv: "a72019de6365e9f9f384794139376e28"
server_key_version: "encryptionkey@0"
I20260929 03:01:38.950855 7513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "b3e08ff429874e0c969fb26d16786b94"
format_stamp: "Formatted at 2026-09-29 03:01:38 on dist-test-slave-530n"
server_key: "98884381badc41cabfc3988161bbd5dc"
server_key_iv: "a72019de6365e9f9f384794139376e28"
server_key_version: "encryptionkey@0"
I20260929 03:01:38.956013 7513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 03:01:38.959481 7527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:38.960907 7513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:38.961059 7513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "b3e08ff429874e0c969fb26d16786b94"
format_stamp: "Formatted at 2026-09-29 03:01:38 on dist-test-slave-530n"
server_key: "98884381badc41cabfc3988161bbd5dc"
server_key_iv: "a72019de6365e9f9f384794139376e28"
server_key_version: "encryptionkey@0"
I20260929 03:01:38.961269 7513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:38.966893 7513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:38.967618 7513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:38.967887 7513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:38.984553 7513 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:39025
I20260929 03:01:38.984586 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:39025 every 8 connection(s)
I20260929 03:01:38.985849 7513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:38.989480 7579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:38.990154 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7513
I20260929 03:01:38.990351 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:38.991184 2870 external_mini_cluster.cc:1428] Setting key b2a269ab90f66be095e9b2ab4b91fff6
I20260929 03:01:38.999315 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:38.993326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.2.205.190:39025)
0929 03:01:38.994311 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:38.994321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:38.994369 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:01:38.995873 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:38.995894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:38.996322 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:38.996691 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:38.997731 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:38.997738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:38.997751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:38.997771 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:38.997833 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:38.997849 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:38.998046 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:38.998175 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:01:38.998418 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260929 03:01:39.002471 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: Bootstrap starting.
I20260929 03:01:39.006053 7579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:39.007120 7579 log.cc:824] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:39.010300 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: No bootstrap required, opened a new log
I20260929 03:01:39.017728 7579 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } }
I20260929 03:01:39.018146 7579 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:39.018249 7579 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e08ff429874e0c969fb26d16786b94, State: Initialized, Role: FOLLOWER
I20260929 03:01:39.018950 7579 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } }
I20260929 03:01:39.019168 7579 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:39.019296 7579 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:39.019450 7579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:39.021821 7579 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } }
I20260929 03:01:39.022403 7579 leader_election.cc:302] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e08ff429874e0c969fb26d16786b94; no voters:
I20260929 03:01:39.023104 7579 leader_election.cc:288] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:39.023274 7584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:39.024394 7584 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [term 1 LEADER]: Becoming Leader. State: Replica: b3e08ff429874e0c969fb26d16786b94, State: Running, Role: LEADER
I20260929 03:01:39.024995 7584 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } }
I20260929 03:01:39.025645 7579 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:39.027719 7586 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e08ff429874e0c969fb26d16786b94. Latest consensus state: current_term: 1 leader_uuid: "b3e08ff429874e0c969fb26d16786b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } } }
I20260929 03:01:39.027842 7585 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e08ff429874e0c969fb26d16786b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e08ff429874e0c969fb26d16786b94" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39025 } } }
I20260929 03:01:39.028122 7585 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:39.028126 7586 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:39.029870 7592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:39.035346 7592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:39.045913 7592 catalog_manager.cc:1347] Generated new cluster ID: 1410c72961ab435e871e6e3b151b2672
I20260929 03:01:39.046046 7592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:39.066509 7592 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:39.067028 7592 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:39.076143 7592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: Generated new TSK 0
I20260929 03:01:39.077301 7592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:01:39.083063 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--builtin_ntp_servers=127.2.205.148:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:01:39.408675 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:39.409022 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:39.409144 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:39.416466 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:39.416601 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:39.416818 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:39.426649 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:39.427985 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:39.429953 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:39.443198 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:39.443158 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:39.443228 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:39.443964 7603 server_base.cc:1034] running on GCE node
I20260929 03:01:39.444588 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:39.445587 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:39.446781 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790650899446730 us; error 44 us; skew 500 ppm
I20260929 03:01:39.447085 7603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:39.449302 7603 webserver.cc:460] Webserver started at http://127.2.205.129:36321/ using document root <none> and password file <none>
I20260929 03:01:39.449983 7603 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:39.450083 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:39.450388 7603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:39.453198 7603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "b34260b94bdc4ff99cc07c593ac7701c"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "fdb5c2cc1062a35747954cea3fe55a5a"
server_key_iv: "30ebab59a104794ce28cba2f4c299c7d"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.453946 7603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "b34260b94bdc4ff99cc07c593ac7701c"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "fdb5c2cc1062a35747954cea3fe55a5a"
server_key_iv: "30ebab59a104794ce28cba2f4c299c7d"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.458941 7603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:39.462320 7617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.463558 7603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:39.463692 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "b34260b94bdc4ff99cc07c593ac7701c"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "fdb5c2cc1062a35747954cea3fe55a5a"
server_key_iv: "30ebab59a104794ce28cba2f4c299c7d"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.463891 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:39.469254 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:39.470000 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:39.470224 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:39.471097 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:39.472393 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:39.472473 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.472572 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:39.472642 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.502545 7603 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37455
I20260929 03:01:39.502595 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37455 every 8 connection(s)
I20260929 03:01:39.503948 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:39.512503 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:39.506773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:39961 (local address 127.2.205.190:39025)
0929 03:01:39.507204 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:39.507209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:39.508490 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 03:01:39.510265 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:39.510270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:39.510339 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:39.510443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:39.511751 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:39.511758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:39.511761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:39.511779 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:39.511821 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:39.511833 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:39.511942 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:39.512138 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:01:39.512209 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260929 03:01:39.513499 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:39.507429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39025 (local address 127.2.205.129:39961)
0929 03:01:39.508326 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:39.508365 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:39.509972 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:39.510702 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:39.510711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:39.510817 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:39.511445 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:39.511447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:39.511487 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:39.511580 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:39.511921 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:39.511932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:39.511940 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:01:39.512206 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 03:01:39.512478 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":244,"threads_started":1}
I20260929 03:01:39.513526 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7603
I20260929 03:01:39.513681 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:39.514425 2870 external_mini_cluster.cc:1428] Setting key d79fe8e63a48897d6dbf66c015cf7070
I20260929 03:01:39.515192 7730 heartbeater.cc:347] Connected to a master server at 127.2.205.190:39025
I20260929 03:01:39.515501 7730 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:39.516103 7730 heartbeater.cc:511] Master 127.2.205.190:39025 requested a full tablet report, sending...
I20260929 03:01:39.517325 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--builtin_ntp_servers=127.2.205.148:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 03:01:39.518301 7544 ts_manager.cc:194] Registered new tserver with Master: b34260b94bdc4ff99cc07c593ac7701c (127.2.205.129:37455)
W20260929 03:01:39.836534 7734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:39.836839 7734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:39.836992 7734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:39.844362 7734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:39.844468 7734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:39.844612 7734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:39.854250 7734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:39.855607 7734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:39.857581 7734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:39.871300 7739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:39.871263 7740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:39.871460 7742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:39.871855 7734 server_base.cc:1034] running on GCE node
I20260929 03:01:39.872620 7734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:39.873649 7734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:39.874861 7734 hybrid_clock.cc:648] HybridClock initialized: now 1790650899874806 us; error 41 us; skew 500 ppm
I20260929 03:01:39.875185 7734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:39.877492 7734 webserver.cc:460] Webserver started at http://127.2.205.130:45039/ using document root <none> and password file <none>
I20260929 03:01:39.878206 7734 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:39.878296 7734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:39.878592 7734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:39.881778 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "3900580ac23c4b178f6450c684fb00ba"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "b71df9390e575ddde17aa8b5fe866729"
server_key_iv: "920fe59a51b1cb4dd06faa55508768f9"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.882714 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "3900580ac23c4b178f6450c684fb00ba"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "b71df9390e575ddde17aa8b5fe866729"
server_key_iv: "920fe59a51b1cb4dd06faa55508768f9"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.888717 7734 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 03:01:39.892719 7748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.894232 7734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:01:39.894399 7734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "3900580ac23c4b178f6450c684fb00ba"
format_stamp: "Formatted at 2026-09-29 03:01:39 on dist-test-slave-530n"
server_key: "b71df9390e575ddde17aa8b5fe866729"
server_key_iv: "920fe59a51b1cb4dd06faa55508768f9"
server_key_version: "encryptionkey@0"
I20260929 03:01:39.894632 7734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:39.900120 7734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:39.900974 7734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:39.901218 7734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:39.902113 7734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:39.903488 7734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:39.903582 7734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.903690 7734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:39.903759 7734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:39.934619 7734 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:33977
I20260929 03:01:39.934697 7860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:33977 every 8 connection(s)
I20260929 03:01:39.935930 7734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:39.943657 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:39.938475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:46993 (local address 127.2.205.190:39025)
0929 03:01:39.938666 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:39.938671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:39.940120 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 03:01:39.941785 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:39.941789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:39.941840 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:39.941928 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:39.943105 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:39.943108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:39.943111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:39.943123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:39.943148 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:39.943158 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:39.943241 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:39.943387 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:01:39.943436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:01:39.944540 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:39.939040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39025 (local address 127.2.205.130:46993)
0929 03:01:39.939973 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:39.940010 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:39.941538 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:39.942115 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:39.942123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:39.942215 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:39.942845 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:39.942847 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:39.942885 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:39.942972 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:39.943223 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:39.943229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:39.943234 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:01:39.943436 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 03:01:39.943631 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260929 03:01:39.945946 7861 heartbeater.cc:347] Connected to a master server at 127.2.205.190:39025
I20260929 03:01:39.946081 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7734
I20260929 03:01:39.946224 7861 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:39.946244 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:39.946663 2870 external_mini_cluster.cc:1428] Setting key 9d37d313247d77f7cb50829fd4ac4d03
I20260929 03:01:39.946815 7861 heartbeater.cc:511] Master 127.2.205.190:39025 requested a full tablet report, sending...
I20260929 03:01:39.948256 7543 ts_manager.cc:194] Registered new tserver with Master: 3900580ac23c4b178f6450c684fb00ba (127.2.205.130:33977)
I20260929 03:01:39.949325 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--builtin_ntp_servers=127.2.205.148:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:01:40.266165 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:40.266427 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:40.266516 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:40.273486 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:40.273573 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:40.273676 7864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:40.285094 7864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:39025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:40.286823 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:40.288638 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:40.301630 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:40.301654 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:40.301709 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:40.302423 7864 server_base.cc:1034] running on GCE node
I20260929 03:01:40.303089 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:40.304077 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:40.305276 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790650900305225 us; error 38 us; skew 500 ppm
I20260929 03:01:40.305635 7864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:40.307989 7864 webserver.cc:460] Webserver started at http://127.2.205.131:35033/ using document root <none> and password file <none>
I20260929 03:01:40.308710 7864 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:40.308840 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:40.309152 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:40.312337 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "4579a97b9ef548e597462fe1dd72a9ca"
format_stamp: "Formatted at 2026-09-29 03:01:40 on dist-test-slave-530n"
server_key: "12f7ec2ed3046b8a53bc7af1b19c9579"
server_key_iv: "3d06c9c5e32943ca0ea36eb4c8c6dd6e"
server_key_version: "encryptionkey@0"
I20260929 03:01:40.313304 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "4579a97b9ef548e597462fe1dd72a9ca"
format_stamp: "Formatted at 2026-09-29 03:01:40 on dist-test-slave-530n"
server_key: "12f7ec2ed3046b8a53bc7af1b19c9579"
server_key_iv: "3d06c9c5e32943ca0ea36eb4c8c6dd6e"
server_key_version: "encryptionkey@0"
I20260929 03:01:40.319113 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:40.323122 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:40.324648 7864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:01:40.324846 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "4579a97b9ef548e597462fe1dd72a9ca"
format_stamp: "Formatted at 2026-09-29 03:01:40 on dist-test-slave-530n"
server_key: "12f7ec2ed3046b8a53bc7af1b19c9579"
server_key_iv: "3d06c9c5e32943ca0ea36eb4c8c6dd6e"
server_key_version: "encryptionkey@0"
I20260929 03:01:40.325090 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:40.330890 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:40.331763 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:40.331986 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:40.332947 7864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:40.334395 7864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:40.334484 7864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:40.334565 7864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:40.334616 7864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:40.363270 7864 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:45825
I20260929 03:01:40.363315 7990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:45825 every 8 connection(s)
I20260929 03:01:40.364553 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:40.367051 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 7864
I20260929 03:01:40.367200 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:40.367719 2870 external_mini_cluster.cc:1428] Setting key 38ddc604f92e41a0799650db9bb6bf53
I20260929 03:01:40.372849 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.367167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:44645 (local address 127.2.205.190:39025)
0929 03:01:40.367403 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.367410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.368953 (+ 1543us) server_negotiation.cc:407] Connection header received
0929 03:01:40.370686 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.370692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.370833 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.370931 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.372126 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.372132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.372137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.372154 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.372185 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.372196 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.372301 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.372501 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.372566 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:01:40.373821 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.367724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39025 (local address 127.2.205.131:44645)
0929 03:01:40.368811 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.368863 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.370353 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.371167 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.371176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.371271 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.371853 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.371855 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.371892 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.371973 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.372275 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.372285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.372294 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.372567 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.372844 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":320,"threads_started":1}
I20260929 03:01:40.375046 7991 heartbeater.cc:347] Connected to a master server at 127.2.205.190:39025
I20260929 03:01:40.375356 7991 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:40.375916 7991 heartbeater.cc:511] Master 127.2.205.190:39025 requested a full tablet report, sending...
I20260929 03:01:40.377245 7543 ts_manager.cc:194] Registered new tserver with Master: 4579a97b9ef548e597462fe1dd72a9ca (127.2.205.131:45825)
I20260929 03:01:40.382903 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:01:40.389882 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.387412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.2.205.190:39025)
0929 03:01:40.387586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.387591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.388074 (+ 483us) server_negotiation.cc:407] Connection header received
0929 03:01:40.388216 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.388220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.388276 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.388375 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.389344 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.389349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.389352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.389368 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.389412 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.389430 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.389539 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.389611 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.389659 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:40.394143 7543 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:40.401109 7543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:40.431475 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.412642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:52186)
0929 03:01:40.426435 (+ 13793us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.426500 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.426780 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.428860 (+ 2080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.428870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.429002 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.430107 (+ 1105us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.430110 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.430142 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.430235 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.430808 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.430812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.430817 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.430971 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.431067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13368,"thread_start_us":13766,"threads_started":1}
I20260929 03:01:40.431633 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.413098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.2.205.130:33977)
0929 03:01:40.425733 (+ 12635us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.425744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.426865 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 03:01:40.427359 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.427369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.427635 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.427794 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.430458 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.430490 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.430506 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.430538 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.430595 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.430613 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.430779 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.431133 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.431254 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12431,"thread_start_us":12341,"threads_started":1}
I20260929 03:01:40.432065 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.412797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.2.205.129:37455)
0929 03:01:40.426217 (+ 13420us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.426227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.426927 (+ 700us) server_negotiation.cc:407] Connection header received
0929 03:01:40.427030 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.427039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.427360 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.427560 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.430479 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.430488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.430504 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.430536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.430598 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.430620 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.430753 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.431622 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.431746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13256,"thread_start_us":12986,"threads_started":1}
I20260929 03:01:40.432384 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.412721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:33977 (local address 127.0.0.1:35834)
0929 03:01:40.426724 (+ 14003us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.426742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.426953 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.429162 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.429168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.429197 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.430155 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.430158 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.430176 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.430234 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.430707 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.430719 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.430728 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.430952 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.431934 (+ 982us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13863,"mutex_wait_us":861,"thread_start_us":13149,"threads_started":1}
I20260929 03:01:40.435092 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.427139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:45825 (local address 127.0.0.1:54474)
0929 03:01:40.431336 (+ 4197us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.431359 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.431478 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.432393 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.432398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.432425 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.433867 (+ 1442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.433870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.433887 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.433946 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.434500 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.434505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.434510 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.434649 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.434701 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4052,"thread_start_us":2606,"threads_started":1}
I20260929 03:01:40.435791 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.427478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.2.205.131:45825)
0929 03:01:40.431101 (+ 3623us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.431110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.431820 (+ 710us) server_negotiation.cc:407] Connection header received
0929 03:01:40.431910 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.431920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.432209 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.432392 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.434126 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.434135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.434151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.434178 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.434242 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.434266 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.434439 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.435326 (+ 887us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.435429 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3444,"thread_start_us":2613,"threads_started":1}
I20260929 03:01:40.436399 7796 tablet_service.cc:1467] Processing CreateTablet for tablet c93ffc9c19f5495486f5baf9eb955146 (DEFAULT_TABLE table=test-table [id=19732b019fb74aaa8c261ff0a7002ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:40.436456 7665 tablet_service.cc:1467] Processing CreateTablet for tablet c93ffc9c19f5495486f5baf9eb955146 (DEFAULT_TABLE table=test-table [id=19732b019fb74aaa8c261ff0a7002ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:40.437888 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93ffc9c19f5495486f5baf9eb955146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:40.437892 7796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93ffc9c19f5495486f5baf9eb955146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:40.439555 7926 tablet_service.cc:1467] Processing CreateTablet for tablet c93ffc9c19f5495486f5baf9eb955146 (DEFAULT_TABLE table=test-table [id=19732b019fb74aaa8c261ff0a7002ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:40.441005 7926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c93ffc9c19f5495486f5baf9eb955146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:40.449644 8009 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Bootstrap starting.
I20260929 03:01:40.451898 8010 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Bootstrap starting.
I20260929 03:01:40.453318 8009 tablet_bootstrap.cc:654] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:40.454623 8009 log.cc:824] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:40.454667 8011 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Bootstrap starting.
I20260929 03:01:40.454974 8010 tablet_bootstrap.cc:654] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:40.455976 8010 log.cc:824] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:40.457613 8011 tablet_bootstrap.cc:654] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:40.457738 8009 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: No bootstrap required, opened a new log
I20260929 03:01:40.458003 8009 ts_tablet_manager.cc:1397] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 03:01:40.458604 8011 log.cc:824] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:40.459134 8010 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: No bootstrap required, opened a new log
I20260929 03:01:40.459371 8010 ts_tablet_manager.cc:1397] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 03:01:40.466436 8011 tablet_bootstrap.cc:492] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: No bootstrap required, opened a new log
I20260929 03:01:40.466666 8011 ts_tablet_manager.cc:1397] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 03:01:40.468124 8009 raft_consensus.cc:348] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.468538 8009 raft_consensus.cc:374] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:40.468628 8009 raft_consensus.cc:729] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34260b94bdc4ff99cc07c593ac7701c, State: Initialized, Role: FOLLOWER
I20260929 03:01:40.469161 8010 raft_consensus.cc:348] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.469311 8009 consensus_queue.cc:260] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.469561 8010 raft_consensus.cc:374] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:40.469650 8010 raft_consensus.cc:729] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3900580ac23c4b178f6450c684fb00ba, State: Initialized, Role: FOLLOWER
I20260929 03:01:40.470258 8010 consensus_queue.cc:260] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.471326 7861 heartbeater.cc:503] Master 127.2.205.190:39025 was elected leader, sending a full tablet report...
I20260929 03:01:40.473642 8010 ts_tablet_manager.cc:1428] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 03:01:40.474505 7730 heartbeater.cc:503] Master 127.2.205.190:39025 was elected leader, sending a full tablet report...
I20260929 03:01:40.475057 8009 ts_tablet_manager.cc:1428] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260929 03:01:40.475010 8011 raft_consensus.cc:348] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.475379 8011 raft_consensus.cc:374] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:40.475466 8011 raft_consensus.cc:729] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4579a97b9ef548e597462fe1dd72a9ca, State: Initialized, Role: FOLLOWER
I20260929 03:01:40.475965 8011 consensus_queue.cc:260] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.477057 7991 heartbeater.cc:503] Master 127.2.205.190:39025 was elected leader, sending a full tablet report...
I20260929 03:01:40.477591 8011 ts_tablet_manager.cc:1428] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
W20260929 03:01:40.506165 7731 tablet.cc:2369] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:40.537881 8015 raft_consensus.cc:482] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:40.538201 8015 raft_consensus.cc:504] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.539990 8015 leader_election.cc:288] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3900580ac23c4b178f6450c684fb00ba (127.2.205.130:33977), 4579a97b9ef548e597462fe1dd72a9ca (127.2.205.131:45825)
I20260929 03:01:40.543195 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.540723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:43037 (local address 127.2.205.130:33977)
0929 03:01:40.541041 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.541048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.541323 (+ 275us) server_negotiation.cc:407] Connection header received
0929 03:01:40.541505 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.541509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.541561 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.541646 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.542623 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.542629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.542632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.542647 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.542676 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.542687 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.542800 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.542868 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.542936 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 03:01:40.543329 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.540449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:59161 (local address 127.2.205.131:45825)
0929 03:01:40.540633 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.540639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.541663 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 03:01:40.541801 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.541805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.541854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.541934 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.542752 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.542756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.542759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.542775 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.542805 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.542816 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.542958 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.543009 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.543072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:01:40.543390 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.540446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:33977 (local address 127.2.205.129:43037)
0929 03:01:40.541191 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.541216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.541374 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.541691 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.541696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.541718 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.542398 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.542402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.542420 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.542469 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.542771 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.542775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.542780 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.542915 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.542964 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":498,"threads_started":1}
I20260929 03:01:40.543846 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.540482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:45825 (local address 127.2.205.129:59161)
0929 03:01:40.541549 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.541563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:40.541676 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:40.541989 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:40.541994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:40.542014 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:40.542549 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:01:40.542552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:40.542565 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:01:40.542601 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:40.542921 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:40.542925 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:40.542929 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:01:40.543058 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:01:40.543107 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":762,"threads_started":1}
I20260929 03:01:40.544441 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93ffc9c19f5495486f5baf9eb955146" candidate_uuid: "b34260b94bdc4ff99cc07c593ac7701c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3900580ac23c4b178f6450c684fb00ba" is_pre_election: true
I20260929 03:01:40.544730 7946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93ffc9c19f5495486f5baf9eb955146" candidate_uuid: "b34260b94bdc4ff99cc07c593ac7701c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4579a97b9ef548e597462fe1dd72a9ca" is_pre_election: true
I20260929 03:01:40.544956 7816 raft_consensus.cc:2454] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b34260b94bdc4ff99cc07c593ac7701c in term 0.
I20260929 03:01:40.545243 7946 raft_consensus.cc:2454] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b34260b94bdc4ff99cc07c593ac7701c in term 0.
I20260929 03:01:40.545596 7620 leader_election.cc:302] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3900580ac23c4b178f6450c684fb00ba, b34260b94bdc4ff99cc07c593ac7701c; no voters:
I20260929 03:01:40.546036 8015 raft_consensus.cc:2781] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:40.546151 8015 raft_consensus.cc:482] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:40.546242 8015 raft_consensus.cc:3037] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:40.548307 8015 raft_consensus.cc:504] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.548906 8015 leader_election.cc:288] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [CANDIDATE]: Term 1 election: Requested vote from peers 3900580ac23c4b178f6450c684fb00ba (127.2.205.130:33977), 4579a97b9ef548e597462fe1dd72a9ca (127.2.205.131:45825)
I20260929 03:01:40.549306 7816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93ffc9c19f5495486f5baf9eb955146" candidate_uuid: "b34260b94bdc4ff99cc07c593ac7701c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3900580ac23c4b178f6450c684fb00ba"
I20260929 03:01:40.549340 7946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c93ffc9c19f5495486f5baf9eb955146" candidate_uuid: "b34260b94bdc4ff99cc07c593ac7701c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4579a97b9ef548e597462fe1dd72a9ca"
I20260929 03:01:40.549508 7816 raft_consensus.cc:3037] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:40.549507 7946 raft_consensus.cc:3037] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:40.552551 7816 raft_consensus.cc:2454] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34260b94bdc4ff99cc07c593ac7701c in term 1.
I20260929 03:01:40.552551 7946 raft_consensus.cc:2454] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34260b94bdc4ff99cc07c593ac7701c in term 1.
I20260929 03:01:40.552950 7620 leader_election.cc:302] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3900580ac23c4b178f6450c684fb00ba, b34260b94bdc4ff99cc07c593ac7701c; no voters:
I20260929 03:01:40.553241 8015 raft_consensus.cc:2781] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:40.553678 8015 raft_consensus.cc:686] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 1 LEADER]: Becoming Leader. State: Replica: b34260b94bdc4ff99cc07c593ac7701c, State: Running, Role: LEADER
I20260929 03:01:40.554138 8015 consensus_queue.cc:237] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } }
I20260929 03:01:40.558154 7543 catalog_manager.cc:5495] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c reported cstate change: term changed from 0 to 1, leader changed from <none> to b34260b94bdc4ff99cc07c593ac7701c (127.2.205.129). New cstate: current_term: 1 leader_uuid: "b34260b94bdc4ff99cc07c593ac7701c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34260b94bdc4ff99cc07c593ac7701c" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:40.592724 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.590603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.2.205.129:37455)
0929 03:01:40.590784 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.590791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.590886 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:01:40.591107 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.591112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.591196 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.591286 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.592108 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.592114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.592118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.592137 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.592171 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.592183 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.592279 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.592382 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.592447 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:01:40.601487 7946 raft_consensus.cc:1261] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 1 FOLLOWER]: Refusing update from remote peer b34260b94bdc4ff99cc07c593ac7701c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:40.601487 7816 raft_consensus.cc:1261] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 1 FOLLOWER]: Refusing update from remote peer b34260b94bdc4ff99cc07c593ac7701c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:40.602648 8020 consensus_queue.cc:1035] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4579a97b9ef548e597462fe1dd72a9ca" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:40.602998 8015 consensus_queue.cc:1035] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3900580ac23c4b178f6450c684fb00ba" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:01:40.616145 7992 tablet.cc:2369] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:40.616607 8025 mvcc.cc:205] Tried to move back new op lower bound from 7334506088852254720 to 7334506088673128448. Current Snapshot: MvccSnapshot[applied={T|T < 7334506088852254720}]
I20260929 03:01:40.630254 7544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:40.630561 7544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:40.634318 7544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3e08ff429874e0c969fb26d16786b94: Sending DeleteTablet for 3 replicas of tablet c93ffc9c19f5495486f5baf9eb955146
I20260929 03:01:40.635429 7796 tablet_service.cc:1514] Processing DeleteTablet for tablet c93ffc9c19f5495486f5baf9eb955146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:40 UTC) from {username='slave'} at 127.0.0.1:35834
I20260929 03:01:40.635524 7926 tablet_service.cc:1514] Processing DeleteTablet for tablet c93ffc9c19f5495486f5baf9eb955146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:40 UTC) from {username='slave'} at 127.0.0.1:54474
I20260929 03:01:40.635535 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet c93ffc9c19f5495486f5baf9eb955146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:40 UTC) from {username='slave'} at 127.0.0.1:52186
I20260929 03:01:40.637918 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7603
I20260929 03:01:40.647656 8035 tablet_replica.cc:330] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: stopping tablet replica
I20260929 03:01:40.648413 8035 raft_consensus.cc:2229] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:40.649173 8035 raft_consensus.cc:2258] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:40.649230 8033 tablet_replica.cc:330] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: stopping tablet replica
I20260929 03:01:40.649689 8033 raft_consensus.cc:2229] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:40.650197 8033 raft_consensus.cc:2258] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:40.651832 8035 ts_tablet_manager.cc:1905] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:40.653271 8034 tablet_replica.cc:330] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c: stopping tablet replica
I20260929 03:01:40.654384 8033 ts_tablet_manager.cc:1905] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:40.654893 8034 raft_consensus.cc:2229] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:40.656157 8034 raft_consensus.cc:2258] T c93ffc9c19f5495486f5baf9eb955146 P b34260b94bdc4ff99cc07c593ac7701c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:40.656705 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.639207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.2.205.129:37455)
0929 03:01:40.639373 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.639379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.653501 (+ 14122us) server_negotiation.cc:407] Connection header received
0929 03:01:40.654031 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.654037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.654108 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.654189 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.655388 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.655394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.655398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.655422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.655456 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.655471 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.656008 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.656061 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.656117 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:01:40.657478 7725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:40.661293 8033 ts_tablet_manager.cc:1918] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:40.661442 8033 log.cc:1196] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/c93ffc9c19f5495486f5baf9eb955146
I20260929 03:01:40.661851 8033 ts_tablet_manager.cc:1939] T c93ffc9c19f5495486f5baf9eb955146 P 3900580ac23c4b178f6450c684fb00ba: Deleting consensus metadata
I20260929 03:01:40.661995 8035 ts_tablet_manager.cc:1918] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:40.662147 8035 log.cc:1196] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/c93ffc9c19f5495486f5baf9eb955146
I20260929 03:01:40.662667 8035 ts_tablet_manager.cc:1939] T c93ffc9c19f5495486f5baf9eb955146 P 4579a97b9ef548e597462fe1dd72a9ca: Deleting consensus metadata
I20260929 03:01:40.663503 7530 catalog_manager.cc:4841] TS 3900580ac23c4b178f6450c684fb00ba (127.2.205.130:33977): tablet c93ffc9c19f5495486f5baf9eb955146 (table test-table [id=19732b019fb74aaa8c261ff0a7002ac7]) successfully deleted
I20260929 03:01:40.664299 7529 catalog_manager.cc:4841] TS 4579a97b9ef548e597462fe1dd72a9ca (127.2.205.131:45825): tablet c93ffc9c19f5495486f5baf9eb955146 (table test-table [id=19732b019fb74aaa8c261ff0a7002ac7]) successfully deleted
I20260929 03:01:40.751101 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7603
I20260929 03:01:40.766438 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7734
W20260929 03:01:40.766554 7529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:37455 (error 108)
I20260929 03:01:40.768375 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.767926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:32992)
0929 03:01:40.768094 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.768229 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:01:40.768581 7529 catalog_manager.cc:4584] TS b34260b94bdc4ff99cc07c593ac7701c (127.2.205.129:37455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c93ffc9c19f5495486f5baf9eb955146: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
I20260929 03:01:40.769677 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.767215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.2.205.130:33977)
0929 03:01:40.767406 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.767412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.767506 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:01:40.767696 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.767701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.767810 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.767908 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.769173 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.769178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.769182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.769198 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.769230 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.769242 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.769358 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.769424 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.769469 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:01:40.770260 7856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:40.822546 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.821900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33000)
0929 03:01:40.822177 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.822346 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 03:01:40.823848 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.823442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33012)
0929 03:01:40.823623 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.823711 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:01:40.859119 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7734
I20260929 03:01:40.872625 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7864
I20260929 03:01:40.875573 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.873585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.2.205.131:45825)
0929 03:01:40.873765 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:40.873770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:40.873790 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:40.873936 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:40.873940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:40.874019 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:40.874104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:40.874969 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:40.874974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:40.874977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:40.874993 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:40.875027 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:40.875039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:40.875152 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:40.875231 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:01:40.875292 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:40.876192 7986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:40.884929 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.884229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33024)
0929 03:01:40.884498 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.884644 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 03:01:40.886083 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.885676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33032)
0929 03:01:40.885839 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.885936 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:01:40.962558 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7864
I20260929 03:01:40.966490 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.966044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33046)
0929 03:01:40.966236 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.966351 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:01:40.967473 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:40.967073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37455 (local address 127.0.0.1:33054)
0929 03:01:40.967291 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:40.967360 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 03:01:40.975427 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 7513
I20260929 03:01:40.976490 7574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:41.081032 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 7513
2026-09-29T03:01:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2558 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:01:41.118321 2870 mini_oidc.cc:152] Starting JWKS server
I20260929 03:01:41.121232 2870 webserver.cc:460] Webserver started at https://127.0.0.1:35253/ using document root <none> and password file <none>
I20260929 03:01:41.121568 2870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:01:41.122287 2870 webserver.cc:460] Webserver started at http://127.0.0.1:36329/ using document root <none> and password file <none>
2026-09-29T03:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:41Z Disabled control of system clock
I20260929 03:01:41.148650 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:32857
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:32857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/testchainca.cert with env {}
W20260929 03:01:41.470606 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:41.470889 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:41.470973 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:41.477998 8047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:41.478111 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:41.478149 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:41.478209 8047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:41.478258 8047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:01:41.478318 8047 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:01:41.478356 8047 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35253/jwks.json
I20260929 03:01:41.488189 8047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:32857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:41.489531 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:41.490754 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:41.501791 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:41.501853 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:41.501837 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:41.502774 8047 server_base.cc:1034] running on GCE node
I20260929 03:01:41.503535 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:41.504962 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:41.506161 8047 hybrid_clock.cc:648] HybridClock initialized: now 1790650901506106 us; error 44 us; skew 500 ppm
I20260929 03:01:41.506493 8047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:41.508605 8047 webserver.cc:460] Webserver started at http://127.2.205.190:40225/ using document root <none> and password file <none>
I20260929 03:01:41.509308 8047 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:41.509424 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:41.509712 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:41.512637 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "baf32822dcb2450f9bab863ccc7d6dea"
format_stamp: "Formatted at 2026-09-29 03:01:41 on dist-test-slave-530n"
server_key: "b427c2cde7a23422b765a389b3fa6617"
server_key_iv: "cc9be056706b2a4050a4cc512fafe255"
server_key_version: "encryptionkey@0"
I20260929 03:01:41.513497 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "baf32822dcb2450f9bab863ccc7d6dea"
format_stamp: "Formatted at 2026-09-29 03:01:41 on dist-test-slave-530n"
server_key: "b427c2cde7a23422b765a389b3fa6617"
server_key_iv: "cc9be056706b2a4050a4cc512fafe255"
server_key_version: "encryptionkey@0"
I20260929 03:01:41.518657 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:41.521996 8061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:41.523288 8047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:41.523451 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "baf32822dcb2450f9bab863ccc7d6dea"
format_stamp: "Formatted at 2026-09-29 03:01:41 on dist-test-slave-530n"
server_key: "b427c2cde7a23422b765a389b3fa6617"
server_key_iv: "cc9be056706b2a4050a4cc512fafe255"
server_key_version: "encryptionkey@0"
I20260929 03:01:41.523659 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:41.559592 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:41.560547 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:41.560885 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:41.578711 8047 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:32857
I20260929 03:01:41.578716 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:32857 every 8 connection(s)
I20260929 03:01:41.580176 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:41.584410 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:41.586597 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8047
I20260929 03:01:41.586799 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:41.587399 2870 external_mini_cluster.cc:1428] Setting key 9e0de8e7cd881e089d4f89a399d04c3d
I20260929 03:01:41.598135 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea: Bootstrap starting.
I20260929 03:01:41.602447 8113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:41.603876 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:41.589412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.2.205.190:32857)
0929 03:01:41.590517 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:41.590532 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:41.590586 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:01:41.592871 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:41.592908 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:41.593533 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:41.594003 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:41.594572 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.596624 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.598091 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.599122 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.599319 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:41.600578 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:41.600589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:41.600610 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:41.601484 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:41.601571 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:41.601599 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:41.601731 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:41.602009 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:01:41.602447 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":290,"threads_started":1}
I20260929 03:01:41.603940 8113 log.cc:824] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:41.607270 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea: No bootstrap required, opened a new log
I20260929 03:01:41.608517 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:01:41.615017 8113 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } }
I20260929 03:01:41.615402 8113 raft_consensus.cc:374] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:41.615495 8113 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf32822dcb2450f9bab863ccc7d6dea, State: Initialized, Role: FOLLOWER
I20260929 03:01:41.616163 8113 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } }
I20260929 03:01:41.616415 8113 raft_consensus.cc:388] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:41.616541 8113 raft_consensus.cc:482] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:41.616693 8113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:41.618803 8113 raft_consensus.cc:504] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } }
I20260929 03:01:41.619289 8113 leader_election.cc:302] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baf32822dcb2450f9bab863ccc7d6dea; no voters:
I20260929 03:01:41.619843 8113 leader_election.cc:288] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:41.619956 8118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:41.620568 8118 raft_consensus.cc:686] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [term 1 LEADER]: Becoming Leader. State: Replica: baf32822dcb2450f9bab863ccc7d6dea, State: Running, Role: LEADER
I20260929 03:01:41.621055 8118 consensus_queue.cc:237] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } }
I20260929 03:01:41.621820 8113 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:41.623999 8120 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader baf32822dcb2450f9bab863ccc7d6dea. Latest consensus state: current_term: 1 leader_uuid: "baf32822dcb2450f9bab863ccc7d6dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } } }
I20260929 03:01:41.624010 8119 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baf32822dcb2450f9bab863ccc7d6dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf32822dcb2450f9bab863ccc7d6dea" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 32857 } } }
I20260929 03:01:41.624496 8119 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:41.624496 8120 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:41.625985 8126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:41.630854 8126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:41.638669 8126 catalog_manager.cc:1347] Generated new cluster ID: 893ccabce54a43afb1c2cfdb6665d880
I20260929 03:01:41.638778 8126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:41.647576 8126 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:41.649351 8126 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:41.659775 8126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baf32822dcb2450f9bab863ccc7d6dea: Generated new TSK 0
I20260929 03:01:41.661128 8126 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:01:41.709486 8143 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:01:41.709692 8143 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:01:41.710474 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:41.681800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.2.205.190:32857)
0929 03:01:41.682024 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:41.682032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:41.682491 (+ 459us) server_negotiation.cc:407] Connection header received
0929 03:01:41.682836 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:41.682840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:41.682909 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:41.683023 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:01:41.683628 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.684373 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.686437 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.686893 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.686990 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:41.687568 (+ 578us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:01:41.709917 (+ 22349us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:01:41.710232 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51564: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 03:01:41.710593 8116 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51564: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:01:41.712436 8142 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:41.681702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:32857 (local address 127.0.0.1:51564)
0929 03:01:41.682383 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:41.682402 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:41.682592 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:41.683078 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:41.683083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:41.683097 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:01:41.683465 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:41.683477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.684498 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.684503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:41.686272 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:41.686288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:41.687056 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:41.687063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:41.687231 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:41.687306 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:01:41.710226 (+ 22920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:01:41.710600 (+ 374us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.205.190:32857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":317,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.205.190:32857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:01:41.714443 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8047
I20260929 03:01:41.715675 8108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:41.810608 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8047
2026-09-29T03:01:41Z chronyd exiting
I20260929 03:01:41.857532 2870 test_util.cc:175] -----------------------------------------------
I20260929 03:01:41.857674 2870 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650859567927-2870-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:01:41.861323 2870 mini_oidc.cc:152] Starting JWKS server
I20260929 03:01:41.863101 2870 webserver.cc:460] Webserver started at https://127.0.0.1:42899/ using document root <none> and password file <none>
I20260929 03:01:41.863286 2870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:01:41.863853 2870 webserver.cc:460] Webserver started at http://127.0.0.1:40989/ using document root <none> and password file <none>
2026-09-29T03:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:41Z Disabled control of system clock
I20260929 03:01:41.888120 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46455
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:46455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42899/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/testchainca.cert with env {}
W20260929 03:01:42.205432 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:42.205781 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:42.205920 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:42.213480 8150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:42.213656 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:42.213726 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:42.213807 8150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:42.213868 8150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:01:42.213941 8150 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:01:42.213991 8150 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42899/jwks.json
I20260929 03:01:42.224632 8150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:46455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42899/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:42.226073 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:42.227413 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:42.239034 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:42.239060 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:42.238996 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:42.239617 8150 server_base.cc:1034] running on GCE node
I20260929 03:01:42.240433 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:42.241918 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:42.243126 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790650902243070 us; error 39 us; skew 500 ppm
I20260929 03:01:42.243472 8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:42.245718 8150 webserver.cc:460] Webserver started at http://127.2.205.190:42907/ using document root <none> and password file <none>
I20260929 03:01:42.246392 8150 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:42.246495 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:42.246803 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:42.249637 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "67e68939ffff453ea1fb64bd216723c3"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "bc7969134211104a0bc96b1c0a9adc3b"
server_key_iv: "bb9950a218d599db8089fe27ea4cf93a"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.250398 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "67e68939ffff453ea1fb64bd216723c3"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "bc7969134211104a0bc96b1c0a9adc3b"
server_key_iv: "bb9950a218d599db8089fe27ea4cf93a"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.255250 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:42.258567 8164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:42.259768 8150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:42.259923 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "67e68939ffff453ea1fb64bd216723c3"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "bc7969134211104a0bc96b1c0a9adc3b"
server_key_iv: "bb9950a218d599db8089fe27ea4cf93a"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.260102 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:42.272065 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:42.272819 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:42.273087 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:42.290716 8150 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:46455
I20260929 03:01:42.290740 8215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:46455 every 8 connection(s)
I20260929 03:01:42.292091 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:42.295874 8216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:42.296742 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8150
I20260929 03:01:42.296974 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:42.297602 2870 external_mini_cluster.cc:1428] Setting key 96534339683b3a6021e3413620b0f611
I20260929 03:01:42.308007 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3: Bootstrap starting.
I20260929 03:01:42.310402 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:42.299468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.2.205.190:46455)
0929 03:01:42.300445 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:42.300457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:42.300505 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:01:42.302070 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:42.302093 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:42.302607 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:42.302967 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:42.303526 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.304784 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.306043 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.306700 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.306870 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:42.308067 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:42.308075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:42.308092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:42.308645 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:42.308720 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:42.308740 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:42.308876 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:42.309150 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:01:42.309473 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":302,"threads_started":1}
I20260929 03:01:42.311223 8216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:42.312454 8216 log.cc:824] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:42.314332 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:01:42.315303 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3: No bootstrap required, opened a new log
I20260929 03:01:42.322667 8216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } }
I20260929 03:01:42.323057 8216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:42.323161 8216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e68939ffff453ea1fb64bd216723c3, State: Initialized, Role: FOLLOWER
I20260929 03:01:42.323753 8216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } }
I20260929 03:01:42.323978 8216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:42.324103 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:42.324255 8216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:42.326416 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } }
I20260929 03:01:42.326889 8216 leader_election.cc:302] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67e68939ffff453ea1fb64bd216723c3; no voters:
I20260929 03:01:42.327485 8216 leader_election.cc:288] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:42.327595 8221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:42.328349 8221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [term 1 LEADER]: Becoming Leader. State: Replica: 67e68939ffff453ea1fb64bd216723c3, State: Running, Role: LEADER
I20260929 03:01:42.328930 8221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } }
I20260929 03:01:42.329596 8216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:42.331502 8223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e68939ffff453ea1fb64bd216723c3. Latest consensus state: current_term: 1 leader_uuid: "67e68939ffff453ea1fb64bd216723c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } } }
I20260929 03:01:42.331913 8223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:42.331811 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67e68939ffff453ea1fb64bd216723c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e68939ffff453ea1fb64bd216723c3" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 46455 } } }
I20260929 03:01:42.332053 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:42.333776 8229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:42.338306 8229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:42.346680 8229 catalog_manager.cc:1347] Generated new cluster ID: 902f9ecc1019477590da407d510ad17a
I20260929 03:01:42.346794 8229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:42.370921 8229 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:42.372872 8229 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:42.380395 8229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67e68939ffff453ea1fb64bd216723c3: Generated new TSK 0
I20260929 03:01:42.381342 8229 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:01:42.414412 8246 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:01:42.414531 8246 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:01:42.415827 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:42.395145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.2.205.190:46455)
0929 03:01:42.395353 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:42.395359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:42.395839 (+ 480us) server_negotiation.cc:407] Connection header received
0929 03:01:42.396052 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:42.396058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:42.396170 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:42.396357 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:01:42.396827 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.398045 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.399520 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.400215 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.400357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:42.400917 (+ 560us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:01:42.415123 (+ 14206us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:01:42.415521 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 03:01:42.415949 8245 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:42.395075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:46455 (local address 127.0.0.1:54844)
0929 03:01:42.395718 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:42.395737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:42.395881 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:42.396355 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:42.396358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:42.396367 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:01:42.396648 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:42.396657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.398211 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.398215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:42.399370 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:42.399384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:42.400399 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:42.400405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:42.400543 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:42.400581 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:01:42.415407 (+ 14826us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:01:42.415634 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.205.190:46455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":358,"threads_started":1}
W20260929 03:01:42.415971 8219 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 03:01:42.417320 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8150
I20260929 03:01:42.418646 8211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:42.511919 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8150
2026-09-29T03:01:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (702 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:01:42.563742 2870 mini_oidc.cc:152] Starting JWKS server
I20260929 03:01:42.565752 2870 webserver.cc:460] Webserver started at https://127.0.0.1:34311/ using document root <none> and password file <none>
I20260929 03:01:42.566004 2870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:01:42.566643 2870 webserver.cc:460] Webserver started at http://127.0.0.1:34359/ using document root <none> and password file <none>
2026-09-29T03:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:42Z Disabled control of system clock
I20260929 03:01:42.592437 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40569
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40569
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:01:42.914588 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:42.914904 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:42.914991 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:42.922322 8253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:42.922439 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:42.922477 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:42.922531 8253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:42.922572 8253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:01:42.922632 8253 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:01:42.922672 8253 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34311/jwks.json
I20260929 03:01:42.932068 8253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:42.933499 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:42.935534 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:42.947854 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:42.947885 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:42.947847 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:42.948741 8253 server_base.cc:1034] running on GCE node
I20260929 03:01:42.949558 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:42.950937 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:42.952144 8253 hybrid_clock.cc:648] HybridClock initialized: now 1790650902952103 us; error 27 us; skew 500 ppm
I20260929 03:01:42.952493 8253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:42.954612 8253 webserver.cc:460] Webserver started at http://127.2.205.190:34519/ using document root <none> and password file <none>
I20260929 03:01:42.955233 8253 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:42.955312 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:42.955614 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:42.958429 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "01d1d903f9bb4673a1cfc10cb59f7938"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "ad571bbcec7dcfba43ef457946cdaa19"
server_key_iv: "3abfe2e76245f8fb0292925f9bdf3ffd"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.959165 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "01d1d903f9bb4673a1cfc10cb59f7938"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "ad571bbcec7dcfba43ef457946cdaa19"
server_key_iv: "3abfe2e76245f8fb0292925f9bdf3ffd"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.964129 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 03:01:42.967655 8267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:42.968995 8253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:01:42.969132 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "01d1d903f9bb4673a1cfc10cb59f7938"
format_stamp: "Formatted at 2026-09-29 03:01:42 on dist-test-slave-530n"
server_key: "ad571bbcec7dcfba43ef457946cdaa19"
server_key_iv: "3abfe2e76245f8fb0292925f9bdf3ffd"
server_key_version: "encryptionkey@0"
I20260929 03:01:42.969297 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:42.984515 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:42.985452 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:42.985724 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:43.003289 8253 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:40569
I20260929 03:01:43.003328 8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:40569 every 8 connection(s)
I20260929 03:01:43.004686 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:43.008777 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:43.011440 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8253
I20260929 03:01:43.011667 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:43.012420 2870 external_mini_cluster.cc:1428] Setting key 877d3196c657e59069c56f536ce78033
I20260929 03:01:43.021955 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938: Bootstrap starting.
I20260929 03:01:43.025925 8319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:43.027109 8319 log.cc:824] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:43.027400 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:43.014643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.2.205.190:40569)
0929 03:01:43.015929 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:43.015942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:43.015993 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:01:43.017865 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:43.017889 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:43.018424 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:43.018796 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:43.019407 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.020716 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.022553 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.023286 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.023430 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.024711 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:43.024719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:43.024735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:43.025376 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:43.025476 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:43.025499 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:43.025590 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:43.025885 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:01:43.026320 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":420,"threads_started":1}
I20260929 03:01:43.030409 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938: No bootstrap required, opened a new log
I20260929 03:01:43.031342 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:01:43.037837 8319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } }
I20260929 03:01:43.038266 8319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:43.038373 8319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d1d903f9bb4673a1cfc10cb59f7938, State: Initialized, Role: FOLLOWER
I20260929 03:01:43.038986 8319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } }
I20260929 03:01:43.039215 8319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:43.039347 8319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:43.039501 8319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:43.041741 8319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } }
I20260929 03:01:43.042286 8319 leader_election.cc:302] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01d1d903f9bb4673a1cfc10cb59f7938; no voters:
I20260929 03:01:43.042925 8319 leader_election.cc:288] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:43.043059 8324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:43.043675 8324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [term 1 LEADER]: Becoming Leader. State: Replica: 01d1d903f9bb4673a1cfc10cb59f7938, State: Running, Role: LEADER
I20260929 03:01:43.044250 8324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } }
I20260929 03:01:43.045275 8319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:43.047042 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d1d903f9bb4673a1cfc10cb59f7938. Latest consensus state: current_term: 1 leader_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } } }
I20260929 03:01:43.047410 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:43.047250 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d1d903f9bb4673a1cfc10cb59f7938" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40569 } } }
I20260929 03:01:43.047593 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:43.049793 8331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:43.054802 8331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:43.063536 8331 catalog_manager.cc:1347] Generated new cluster ID: ce24650a74f34bc7a8f3252a8e7073f0
I20260929 03:01:43.063632 8331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:43.072288 8331 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:43.074105 8331 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:43.086012 8331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d1d903f9bb4673a1cfc10cb59f7938: Generated new TSK 0
I20260929 03:01:43.086969 8331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:01:43.116946 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:43.108118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.2.205.190:40569)
0929 03:01:43.108277 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:43.108282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:43.108667 (+ 385us) server_negotiation.cc:407] Connection header received
0929 03:01:43.108941 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:43.108948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:43.109061 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:43.109242 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:01:43.109669 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.110803 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.112190 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.112945 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.113078 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.113655 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:01:43.116191 (+ 2536us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:01:43.116580 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 03:01:43.117084 8322 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:01:43.118685 8348 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:43.107920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40569 (local address 127.0.0.1:53284)
0929 03:01:43.108558 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:43.108574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:43.108708 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:43.109255 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:43.109258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:43.109269 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:01:43.109520 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:43.109528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.110942 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.110946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:43.112045 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:43.112057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.113175 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.113181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:43.113329 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.113367 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:01:43.116433 (+ 3066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:01:43.118233 (+ 1800us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.205.190:40569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1457,"thread_start_us":312,"threads_started":1}
I20260929 03:01:43.119781 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8253
I20260929 03:01:43.121060 8314 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 03:01:43.128345 8349 webserver.cc:533] Webserver: sslize failed:
E20260929 03:01:43.128441 8349 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:01:43.213083 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8253
2026-09-29T03:01:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (701 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:43Z Disabled control of system clock
I20260929 03:01:43.289551 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40541
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:40023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40541
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/testchainca.cert with env {}
W20260929 03:01:43.608505 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:43.608878 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:43.609017 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:43.616398 8354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:43.616509 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:43.616546 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:43.616623 8354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:43.616672 8354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:01:43.616784 8354 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:01:43.616842 8354 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:01:43.627087 8354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:43.628435 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:43.629771 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:43.641317 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:43.641465 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:43.641465 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:43.642627 8354 server_base.cc:1034] running on GCE node
I20260929 03:01:43.643461 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:43.644961 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:43.646163 8354 hybrid_clock.cc:648] HybridClock initialized: now 1790650903646113 us; error 44 us; skew 500 ppm
I20260929 03:01:43.646504 8354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:01:43.648574 8354 webserver.cc:460] Webserver started at http://127.2.205.190:44005/ using document root <none> and password file <none>
I20260929 03:01:43.649279 8354 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:43.649379 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:43.649667 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:43.652509 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "7b352b09b74e43fea65a31bdce90808d"
format_stamp: "Formatted at 2026-09-29 03:01:43 on dist-test-slave-530n"
server_key: "33a18bcc66e96bc8ca4bf49115cd4e44"
server_key_iv: "29f225f0c40ba18534661e43ccc2e350"
server_key_version: "encryptionkey@0"
I20260929 03:01:43.653290 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "7b352b09b74e43fea65a31bdce90808d"
format_stamp: "Formatted at 2026-09-29 03:01:43 on dist-test-slave-530n"
server_key: "33a18bcc66e96bc8ca4bf49115cd4e44"
server_key_iv: "29f225f0c40ba18534661e43ccc2e350"
server_key_version: "encryptionkey@0"
I20260929 03:01:43.658030 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 03:01:43.661176 8368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:43.662381 8354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260929 03:01:43.662524 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "7b352b09b74e43fea65a31bdce90808d"
format_stamp: "Formatted at 2026-09-29 03:01:43 on dist-test-slave-530n"
server_key: "33a18bcc66e96bc8ca4bf49115cd4e44"
server_key_iv: "29f225f0c40ba18534661e43ccc2e350"
server_key_version: "encryptionkey@0"
I20260929 03:01:43.662696 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:43.680013 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:43.680871 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:43.681133 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:43.698686 8354 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:40541
I20260929 03:01:43.698707 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:40541 every 8 connection(s)
I20260929 03:01:43.700155 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:43.704216 8420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:43.708104 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8354
I20260929 03:01:43.708288 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:43.708987 2870 external_mini_cluster.cc:1428] Setting key 198ba1e64cc341e2e061debb3fe7646e
I20260929 03:01:43.716048 8420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d: Bootstrap starting.
I20260929 03:01:43.719600 8420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:43.720820 8420 log.cc:824] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:43.723839 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:43.711067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.2.205.190:40541)
0929 03:01:43.712161 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:43.712175 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:43.712221 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:01:43.714544 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:43.714573 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:43.715135 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:43.715559 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:43.716203 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.717666 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.719028 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.719780 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.719990 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.721224 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:43.721232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:43.721248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:01:43.722050 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:43.722169 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:01:43.722198 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:43.722331 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:43.722572 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 03:01:43.722913 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":345,"threads_started":1}
I20260929 03:01:43.724032 8420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d: No bootstrap required, opened a new log
I20260929 03:01:43.727963 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:01:43.732359 8420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } }
I20260929 03:01:43.732779 8420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:43.732877 8420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b352b09b74e43fea65a31bdce90808d, State: Initialized, Role: FOLLOWER
I20260929 03:01:43.733475 8420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } }
I20260929 03:01:43.733691 8420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:43.733815 8420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:43.733973 8420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:43.736182 8420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } }
I20260929 03:01:43.736702 8420 leader_election.cc:302] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b352b09b74e43fea65a31bdce90808d; no voters:
I20260929 03:01:43.737301 8420 leader_election.cc:288] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:43.737433 8425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:43.738088 8425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [term 1 LEADER]: Becoming Leader. State: Replica: 7b352b09b74e43fea65a31bdce90808d, State: Running, Role: LEADER
I20260929 03:01:43.738651 8425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } }
I20260929 03:01:43.739306 8420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:43.741323 8427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b352b09b74e43fea65a31bdce90808d. Latest consensus state: current_term: 1 leader_uuid: "7b352b09b74e43fea65a31bdce90808d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } } }
I20260929 03:01:43.741536 8426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b352b09b74e43fea65a31bdce90808d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b352b09b74e43fea65a31bdce90808d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40541 } } }
I20260929 03:01:43.741753 8427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:43.741851 8426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:43.742996 8432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:43.748265 8432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:43.757081 8432 catalog_manager.cc:1347] Generated new cluster ID: dda675ca06a044eba2bbf4b206ad516e
I20260929 03:01:43.757180 8432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:43.774593 8432 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:43.775735 8432 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:43.792068 8432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b352b09b74e43fea65a31bdce90808d: Generated new TSK 0
I20260929 03:01:43.793108 8432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:01:44.708732 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:43.804688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.2.205.190:40541)
0929 03:01:43.804901 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:43.804911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:43.805297 (+ 386us) server_negotiation.cc:407] Connection header received
0929 03:01:43.805541 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:43.805545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:43.805625 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:43.805770 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:01:43.806260 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.807050 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.808436 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.808944 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.809059 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.809560 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:01:44.708029 (+898469us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:01:44.708435 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39224: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 03:01:44.708889 8423 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39224: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:01:44.709010 8449 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:43.804568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40541 (local address 127.0.0.1:39224)
0929 03:01:43.805189 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:43.805204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:43.805352 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:43.805803 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:43.805807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:43.805820 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:01:43.806104 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:43.806115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.807197 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.807201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:43.808293 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:43.808305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:43.809131 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:43.809138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:43.809267 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:43.809303 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:01:44.708354 (+899051us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:01:44.708617 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.205.190:40541: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":339,"threads_started":1}
I20260929 03:01:44.710526 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8354
I20260929 03:01:44.711751 8415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:44.814380 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8354
2026-09-29T03:01:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1581 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (419 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (406 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:45 dist-test-slave-530n krb5kdc[8464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:45 dist-test-slave-530n krb5kdc[8464](info): set up 2 sockets
Sep 29 03:01:45 dist-test-slave-530n krb5kdc[8464](info): commencing operation
krb5kdc: starting...
W20260929 03:01:46.637666 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:46 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:01:46.837206 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45353
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:47.155234 8477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:47.155521 8477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:47.155633 8477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:47.163075 8477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:47.163182 8477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:47.163233 8477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:47.163327 8477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:47.163383 8477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:47.173089 8477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:47.174382 8477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:47.175648 8477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:47.187543 8482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:47.187575 8483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:47.187623 8485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:47.188439 8477 server_base.cc:1034] running on GCE node
I20260929 03:01:47.189266 8477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:47.190724 8477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:47.191946 8477 hybrid_clock.cc:648] HybridClock initialized: now 1790650907191898 us; error 35 us; skew 500 ppm
I20260929 03:01:47.192296 8477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:47 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:47.195485 8477 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:01:47.197023 8477 webserver.cc:460] Webserver started at http://127.2.205.190:40887/ using document root <none> and password file <none>
I20260929 03:01:47.197719 8477 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:47.197825 8477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:47.198108 8477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:47.201216 8477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "e3ea6a8f3a02495bb46d9dc440c34055"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "17a06b524941c3592975a5836fc50314"
server_key_iv: "fd6b5b262ae6431edb054fd6889017a6"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.201993 8477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "e3ea6a8f3a02495bb46d9dc440c34055"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "17a06b524941c3592975a5836fc50314"
server_key_iv: "fd6b5b262ae6431edb054fd6889017a6"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.207370 8477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:01:47.210944 8492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:47.212445 8477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:01:47.212600 8477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "e3ea6a8f3a02495bb46d9dc440c34055"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "17a06b524941c3592975a5836fc50314"
server_key_iv: "fd6b5b262ae6431edb054fd6889017a6"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.212811 8477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:47.225216 8477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:47.229760 8477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:47.230023 8477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:47.249312 8477 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:45353
I20260929 03:01:47.249341 8543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:45353 every 8 connection(s)
I20260929 03:01:47.250690 8477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:47.254392 8544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:47.256047 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8477
I20260929 03:01:47.256210 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:47.256865 2870 external_mini_cluster.cc:1428] Setting key 3d8a4178636be973035f8fa945ef293e
I20260929 03:01:47.265646 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055: Bootstrap starting.
I20260929 03:01:47.269194 8544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:47.270363 8544 log.cc:824] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:01:47 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:47.273447 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055: No bootstrap required, opened a new log
I20260929 03:01:47.279790 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:47.258715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.2.205.190:45353)
0929 03:01:47.259451 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:47.259464 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:47.259512 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:01:47.261206 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:47.261229 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:47.261609 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:47.261929 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:47.263535 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.264848 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.266522 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.267179 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.267300 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:47.271794 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:47.271805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:47.271822 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:47.271869 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:47.275491 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:47.276096 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:47.276105 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:47.276113 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:47.276321 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:47.276624 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:47.276629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:47.276633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:47.277336 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:47.277638 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:47.277920 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 03:01:47.278367 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260929 03:01:47.283432 8544 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } }
I20260929 03:01:47.283769 8544 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:47.283851 8544 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ea6a8f3a02495bb46d9dc440c34055, State: Initialized, Role: FOLLOWER
I20260929 03:01:47.284390 8544 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } }
I20260929 03:01:47.284586 8544 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:47.284678 8544 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:47.284860 8544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:47.287173 8544 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } }
I20260929 03:01:47.287637 8544 leader_election.cc:302] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ea6a8f3a02495bb46d9dc440c34055; no voters:
I20260929 03:01:47.288157 8544 leader_election.cc:288] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:47.288297 8549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:47.288947 8549 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [term 1 LEADER]: Becoming Leader. State: Replica: e3ea6a8f3a02495bb46d9dc440c34055, State: Running, Role: LEADER
I20260929 03:01:47.289491 8549 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } }
I20260929 03:01:47.290218 8544 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:47.292353 8551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ea6a8f3a02495bb46d9dc440c34055. Latest consensus state: current_term: 1 leader_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } } }
I20260929 03:01:47.292351 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ea6a8f3a02495bb46d9dc440c34055" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45353 } } }
I20260929 03:01:47.292897 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:47.292887 8551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:47.295814 8554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:47.301177 8554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:47.310016 8554 catalog_manager.cc:1347] Generated new cluster ID: 819442f523394232a6c8e3a7b2f1b071
I20260929 03:01:47.310139 8554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:47.318889 8554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:47.320133 8554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:47.327350 8554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ea6a8f3a02495bb46d9dc440c34055: Generated new TSK 0
I20260929 03:01:47.328125 8554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:01:47.385457 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:47.708174 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:47.708493 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:47.708587 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:47.716171 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:47.716313 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:47.716440 8570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:47.726678 8570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:47.727982 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:47.729346 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:47.742367 8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:47.742352 8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:47.742343 8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:47.743367 8570 server_base.cc:1034] running on GCE node
I20260929 03:01:47.744046 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:47.745134 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:47.746376 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790650907746323 us; error 49 us; skew 500 ppm
I20260929 03:01:47.746678 8570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:47 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:47.749720 8570 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:01:47.751264 8570 webserver.cc:460] Webserver started at http://127.2.205.129:40977/ using document root <none> and password file <none>
I20260929 03:01:47.751958 8570 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:47.752074 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:47.752377 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:47.755394 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "4219e39ca6504db4952776d34bb6d823"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "1128c21c335b0fdfda9757fc99fa469e"
server_key_iv: "695e9b0b681d8e3fcc5449f2d8eb8200"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.756168 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "4219e39ca6504db4952776d34bb6d823"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "1128c21c335b0fdfda9757fc99fa469e"
server_key_iv: "695e9b0b681d8e3fcc5449f2d8eb8200"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.761111 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 03:01:47.764333 8585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:47.765694 8570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:47.765838 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "4219e39ca6504db4952776d34bb6d823"
format_stamp: "Formatted at 2026-09-29 03:01:47 on dist-test-slave-530n"
server_key: "1128c21c335b0fdfda9757fc99fa469e"
server_key_iv: "695e9b0b681d8e3fcc5449f2d8eb8200"
server_key_version: "encryptionkey@0"
I20260929 03:01:47.766016 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:47.811553 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:47.815977 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:47.816289 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:47.817353 8570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:47.818733 8570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:47.818816 8570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:47.818936 8570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:47.818990 8570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:47.848990 8570 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:35149
I20260929 03:01:47.849038 8697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:35149 every 8 connection(s)
I20260929 03:01:47.850211 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:47.856465 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8570
I20260929 03:01:47.856647 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:47.857255 2870 external_mini_cluster.cc:1428] Setting key 3b02e836197125f5f0bd7dd6b3d06cb4
Sep 29 03:01:47 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:47.869570 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:47.852683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:58387 (local address 127.2.205.190:45353)
0929 03:01:47.853115 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:47.853120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:47.853981 (+ 861us) server_negotiation.cc:407] Connection header received
0929 03:01:47.855718 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:47.855723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:47.855803 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:47.855906 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:47.857459 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.858286 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.860311 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.860704 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.860822 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:47.864424 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:47.864429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:47.864432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:47.864462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:47.866788 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:47.867430 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:47.867435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:47.867439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:47.867551 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:47.867930 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:47.867935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:47.867937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:47.868178 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:47.868302 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:47.869019 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 03:01:47.869227 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":237,"threads_started":1}
I20260929 03:01:47.870505 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:47.853168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45353 (local address 127.2.205.129:58387)
0929 03:01:47.853833 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:47.853876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:47.855466 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:47.856144 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:47.856152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:47.856680 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:47.857270 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:47.857287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.858451 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.858455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:47.860155 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:47.860171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:47.860940 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:47.860947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:47.861203 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:47.861863 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:47.861890 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:47.864226 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:47.866917 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:47.866926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:47.866942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:47.867325 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:47.867647 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:47.867652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:47.867656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:47.867825 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:47.868293 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:47.868300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:47.868621 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 03:01:47.868905 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 03:01:47.869380 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":250,"threads_started":1}
I20260929 03:01:47.872076 8698 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45353
I20260929 03:01:47.872335 8698 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:47.873095 8698 heartbeater.cc:511] Master 127.2.205.190:45353 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:47.875048 8509 ts_manager.cc:194] Registered new tserver with Master: 4219e39ca6504db4952776d34bb6d823 (127.2.205.129:35149)
I20260929 03:01:47.876505 8509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:58387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:01:47.899858 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:48.213276 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:48.213593 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:48.213728 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:48.221016 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:48.221148 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:48.221302 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:48.231478 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:48.232846 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:48.234165 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:48.249225 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:48.249305 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:48.249809 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:48.250268 8704 server_base.cc:1034] running on GCE node
I20260929 03:01:48.251051 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:48.252251 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:48.253474 8704 hybrid_clock.cc:648] HybridClock initialized: now 1790650908253420 us; error 50 us; skew 500 ppm
I20260929 03:01:48.253832 8704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.257714 8704 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:01:48.261943 8704 webserver.cc:460] Webserver started at http://127.2.205.130:43065/ using document root <none> and password file <none>
I20260929 03:01:48.262658 8704 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:48.262781 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:48.263082 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:48.266417 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "421754f376c340d4a008a71b78fdaff6"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "ca63dac1d9c46f66419fd84f68753f80"
server_key_iv: "e13547820f6f53d63e720e1d3e4a0ffc"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.267279 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "421754f376c340d4a008a71b78fdaff6"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "ca63dac1d9c46f66419fd84f68753f80"
server_key_iv: "e13547820f6f53d63e720e1d3e4a0ffc"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.272886 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:01:48.276692 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.278154 8704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:48.278316 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "421754f376c340d4a008a71b78fdaff6"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "ca63dac1d9c46f66419fd84f68753f80"
server_key_iv: "e13547820f6f53d63e720e1d3e4a0ffc"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.278510 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:48.296983 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:48.301155 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:48.301411 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:48.302379 8704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:48.303687 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:48.303757 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.303841 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:48.303882 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.333861 8704 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:44911
I20260929 03:01:48.333912 8831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:44911 every 8 connection(s)
I20260929 03:01:48.335130 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:48.339121 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8704
I20260929 03:01:48.339293 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:48.339852 2870 external_mini_cluster.cc:1428] Setting key e049f0ebf3ee454c6bb5f265425f15aa
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:48.355958 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.337673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:53945 (local address 127.2.205.190:45353)
0929 03:01:48.337863 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.337868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.339125 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 03:01:48.341354 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.341360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.341426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.341508 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.343465 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.344174 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.346201 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.346628 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.346725 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.350471 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.350476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.350479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.350511 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.353004 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.353783 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.353787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.353789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.353931 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.354347 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.354350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.354352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.354609 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.354710 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.355479 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.355648 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:01:48.357365 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.338135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45353 (local address 127.2.205.130:53945)
0929 03:01:48.338929 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:48.338987 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:48.341069 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:48.341775 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:48.341783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:48.342487 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:48.343268 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:48.343292 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.344307 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.344312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:48.346064 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:48.346078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.346840 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.346847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:48.347099 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.347779 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:48.347812 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:48.350221 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:48.353131 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:48.353140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:48.353156 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:48.353662 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:48.354027 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:48.354031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:48.354035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:48.354258 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:48.354717 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:48.354724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:48.355127 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 03:01:48.355422 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 03:01:48.355938 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260929 03:01:48.358698 8832 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45353
I20260929 03:01:48.359014 8832 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:48.359915 8832 heartbeater.cc:511] Master 127.2.205.190:45353 requested a full tablet report, sending...
I20260929 03:01:48.361352 8509 ts_manager.cc:194] Registered new tserver with Master: 421754f376c340d4a008a71b78fdaff6 (127.2.205.130:44911)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:48.362270 8509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:53945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:01:48.383322 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:48.704425 8837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:48.704766 8837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:48.704916 8837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:48.712332 8837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:48.712483 8837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:48.712641 8837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:48.722862 8837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:48.724285 8837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:48.725665 8837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:48.738940 8842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:48.738880 8845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:48.738904 8843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:48.739818 8837 server_base.cc:1034] running on GCE node
I20260929 03:01:48.740490 8837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:48.741652 8837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:48.742885 8837 hybrid_clock.cc:648] HybridClock initialized: now 1790650908742825 us; error 58 us; skew 500 ppm
I20260929 03:01:48.743199 8837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.746271 8837 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:01:48.747727 8837 webserver.cc:460] Webserver started at http://127.2.205.131:39909/ using document root <none> and password file <none>
I20260929 03:01:48.748420 8837 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:48.748540 8837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:48.748885 8837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:48.751905 8837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "6324fe9a3a4b44d8beea3d9be51077c7"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "5be29183f66128182eae77f92dc48f9b"
server_key_iv: "2b649742e08a1158b06fd6f4075861bd"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.752676 8837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "6324fe9a3a4b44d8beea3d9be51077c7"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "5be29183f66128182eae77f92dc48f9b"
server_key_iv: "2b649742e08a1158b06fd6f4075861bd"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.757711 8837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:48.760841 8852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.762181 8837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:01:48.762331 8837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "6324fe9a3a4b44d8beea3d9be51077c7"
format_stamp: "Formatted at 2026-09-29 03:01:48 on dist-test-slave-530n"
server_key: "5be29183f66128182eae77f92dc48f9b"
server_key_iv: "2b649742e08a1158b06fd6f4075861bd"
server_key_version: "encryptionkey@0"
I20260929 03:01:48.762507 8837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:48.780854 8837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:48.785475 8837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:48.785741 8837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:48.786813 8837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:48.788188 8837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:48.788265 8837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.788369 8837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:48.788429 8837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:48.819554 8837 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41701
I20260929 03:01:48.819589 8964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41701 every 8 connection(s)
I20260929 03:01:48.820880 8837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:48.822798 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 8837
I20260929 03:01:48.822978 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:48.823424 2870 external_mini_cluster.cc:1428] Setting key 71c8bba9dc4b023204845dd307eea5b1
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:01:48.840804 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.823782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48241 (local address 127.2.205.190:45353)
0929 03:01:48.824960 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.824965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.825552 (+ 587us) server_negotiation.cc:407] Connection header received
0929 03:01:48.827404 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.827410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.827521 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.827606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.829213 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.829967 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.831474 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.831881 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.831992 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.835253 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.835257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.835261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.835291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.837786 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.838564 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.838569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.838573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.838705 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.839104 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.839107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.839109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.839368 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.839510 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.840216 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.840444 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053}
I20260929 03:01:48.841722 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.824448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45353 (local address 127.2.205.131:48241)
0929 03:01:48.825394 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:48.825447 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:48.827146 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:48.827853 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:48.827862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:48.828388 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:48.829050 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:48.829070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.830100 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.830104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:48.831343 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:48.831355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.832109 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.832115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:48.832348 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.832982 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:48.833010 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:48.835064 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:48.837916 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:48.837925 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:48.837942 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:48.838452 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:48.838802 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:48.838807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:48.838810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:48.838998 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:48.839497 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:48.839505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:48.839857 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 03:01:48.840102 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 03:01:48.840621 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":383,"threads_started":1}
I20260929 03:01:48.843106 8965 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45353
I20260929 03:01:48.843376 8965 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:48.844066 8965 heartbeater.cc:511] Master 127.2.205.190:45353 requested a full tablet report, sending...
I20260929 03:01:48.845475 8509 ts_manager.cc:194] Registered new tserver with Master: 6324fe9a3a4b44d8beea3d9be51077c7 (127.2.205.131:41701)
I20260929 03:01:48.846261 8509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:48241
I20260929 03:01:48.851140 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/test-admin.keytab.
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.874995 2870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260929 03:01:48.879482 8698 heartbeater.cc:503] Master 127.2.205.190:45353 was elected leader, sending a full tablet report...
W20260929 03:01:48.880043 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:48.877524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.2.205.129:35149)
0929 03:01:48.878201 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.878210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.878239 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:01:48.878301 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.878307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.878545 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.878696 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.879582 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39076: BlockingRecv error: recv got EOF from 127.0.0.1:39076 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
W20260929 03:01:48.880276 2870 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc)
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.882010 2870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.883371 2870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:48.899668 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.885123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.2.205.129:35149)
0929 03:01:48.885323 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.885329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.885533 (+ 204us) server_negotiation.cc:407] Connection header received
0929 03:01:48.885703 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.885709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.885782 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.885865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.886941 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.887931 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.889250 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.889866 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.890011 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.893800 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.893815 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.893833 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.893875 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.896899 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.897620 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.897628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.897631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.897760 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.898144 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.898150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.898152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.898524 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.898807 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.899136 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.899393 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 03:01:48.905810 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:48.903972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.2.205.129:35149)
0929 03:01:48.904175 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.904181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.904461 (+ 280us) server_negotiation.cc:407] Connection header received
0929 03:01:48.904637 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.904641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.904702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.904809 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.905557 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39096: BlockingRecv error: recv got EOF from 127.0.0.1:39096 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 03:01:48.906024 2870 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc)
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.907480 2870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.908720 2870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:48.922783 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.910026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.2.205.129:35149)
0929 03:01:48.910231 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.910236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.910423 (+ 187us) server_negotiation.cc:407] Connection header received
0929 03:01:48.910632 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.910636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.910702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.910785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.911631 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.912288 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.913531 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.913952 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.914049 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.917624 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.917632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.917637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.917693 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.920498 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.921150 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.921154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.921156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.921279 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.921606 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.921609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.921610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.921874 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.922067 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.922289 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.922480 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 03:01:48.927259 8972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:01:48.925564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.2.205.129:35149)
0929 03:01:48.925702 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.925706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.926016 (+ 310us) server_negotiation.cc:407] Connection header received
0929 03:01:48.926152 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.926156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.926239 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.926347 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.927068 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39112: BlockingRecv error: recv got EOF from 127.0.0.1:39112 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 03:01:48.927520 2870 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650859567927-2870-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.929080 2870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:48.930296 2870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:01:48 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:01:48.944000 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.931730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.2.205.129:35149)
0929 03:01:48.931942 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.931946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.932042 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:01:48.932169 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.932173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.932241 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.932318 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.933288 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.934002 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.935233 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.935640 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.935750 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.939242 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.939249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.939253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.939290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.941772 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.942409 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.942413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.942415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.942569 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.942866 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.942869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.942871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.943173 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.943415 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.943578 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.943737 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:01:48.945622 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8570
I20260929 03:01:48.956506 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:48.946284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.2.205.129:35149)
0929 03:01:48.946470 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:48.946477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:48.946735 (+ 258us) server_negotiation.cc:407] Connection header received
0929 03:01:48.946840 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:48.946843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:48.946898 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:48.946970 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:48.947807 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.948493 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.949652 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:48.950060 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:48.950150 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:48.951904 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:48.951909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:48.951912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:48.951938 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:48.954250 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.954845 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.954848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.954851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.954974 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:48.955275 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:48.955279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:48.955281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:48.955529 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:48.955715 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:48.956006 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:01:48.956181 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:01:48.957049 8693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:49.039997 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8570
I20260929 03:01:49.052769 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8704
Sep 29 03:01:49 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:01:49.067328 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:49.053665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.2.205.130:44911)
0929 03:01:49.054182 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:49.054188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:49.054214 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:01:49.054279 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:49.054286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:49.054483 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:49.054618 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:49.055492 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:49.056490 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:49.057785 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:49.058314 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:49.058406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:49.062031 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:49.062043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:49.062058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:49.062092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:49.064715 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:49.065503 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:49.065510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:49.065512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:49.065627 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:49.065937 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:49.065940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:49.065942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:49.066244 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:49.066460 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:49.066821 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 03:01:49.067022 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
I20260929 03:01:49.068502 8827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:49.150133 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8704
I20260929 03:01:49.162559 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8837
Sep 29 03:01:49 dist-test-slave-530n krb5kdc[8464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:01:49.178088 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:49.163664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.2.205.131:41701)
0929 03:01:49.164286 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:49.164293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:49.164322 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:01:49.164389 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:49.164397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:49.164604 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:49.164735 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:49.165783 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:49.166876 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:49.168089 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:49.168632 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:49.168727 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:49.172633 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:49.172644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:49.172666 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:49.172713 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:49.175568 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:49.176213 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:49.176219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:49.176222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:49.176338 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:49.176673 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:49.176677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:49.176679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:49.177071 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:49.177302 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:49.177518 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:01:49.177752 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
I20260929 03:01:49.179389 8960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:49.264788 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8837
I20260929 03:01:49.277237 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 8477
I20260929 03:01:49.278415 8539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:49.379223 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 8477
2026-09-29T03:01:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3742 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:01:49 dist-test-slave-530n krb5kdc[8992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:01:49 dist-test-slave-530n krb5kdc[8992](info): set up 2 sockets
Sep 29 03:01:49 dist-test-slave-530n krb5kdc[8992](info): commencing operation
krb5kdc: starting...
W20260929 03:01:50.373473 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:01:50 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:01:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab.
Entry for principal oryx/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab.
I20260929 03:01:50.571943 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40357
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab
--principal=oryx/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:50.890908 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:50.891198 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:50.891312 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:50.898394 9005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:50.898496 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:50.898550 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:50.898625 9005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:50.898684 9005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:50.908310 9005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab
--principal=oryx/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:50.909713 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:50.911158 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:50.923183 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:50.923240 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:50.923218 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:50.923887 9005 server_base.cc:1034] running on GCE node
I20260929 03:01:50.924841 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:50.926272 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:50.927479 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790650910927427 us; error 44 us; skew 500 ppm
I20260929 03:01:50.927820 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:50 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:50.934075 9005 init.cc:377] Logged in from keytab as oryx/127.2.205.190@KRBTEST.COM (short username oryx)
I20260929 03:01:50.935550 9005 webserver.cc:460] Webserver started at http://127.2.205.190:36865/ using document root <none> and password file <none>
I20260929 03:01:50.936244 9005 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:50.936358 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:50.936635 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:50.939606 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "a31052ca45b74cbc827b973e04cec020"
format_stamp: "Formatted at 2026-09-29 03:01:50 on dist-test-slave-530n"
server_key: "b81f5801f716fe143a0c8be0f39d3158"
server_key_iv: "46ba060332f3c050ee43a072f6d5f8d3"
server_key_version: "encryptionkey@0"
I20260929 03:01:50.940397 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "a31052ca45b74cbc827b973e04cec020"
format_stamp: "Formatted at 2026-09-29 03:01:50 on dist-test-slave-530n"
server_key: "b81f5801f716fe143a0c8be0f39d3158"
server_key_iv: "46ba060332f3c050ee43a072f6d5f8d3"
server_key_version: "encryptionkey@0"
I20260929 03:01:50.945439 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:01:50.948625 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:50.949985 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:50.950137 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "a31052ca45b74cbc827b973e04cec020"
format_stamp: "Formatted at 2026-09-29 03:01:50 on dist-test-slave-530n"
server_key: "b81f5801f716fe143a0c8be0f39d3158"
server_key_iv: "46ba060332f3c050ee43a072f6d5f8d3"
server_key_version: "encryptionkey@0"
I20260929 03:01:50.950297 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:50.979915 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:50.984244 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:50.984529 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:51.001892 9005 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:40357
I20260929 03:01:51.001945 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:40357 every 8 connection(s)
I20260929 03:01:51.003314 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:01:51.007190 9072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:51.011442 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9005
I20260929 03:01:51.011668 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:01:51.012280 2870 external_mini_cluster.cc:1428] Setting key 9235722bdd3cd43e1026a1cad9b71b72
I20260929 03:01:51.018721 9072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Bootstrap starting.
I20260929 03:01:51.022641 9072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:51.023828 9072 log.cc:824] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:51.026844 9072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: No bootstrap required, opened a new log
Sep 29 03:01:51 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:51.036885 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:51.014412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.2.205.190:40357)
0929 03:01:51.015282 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:51.015297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:51.015347 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:01:51.017213 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:51.017239 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:51.017705 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:51.018088 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:51.019958 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.021412 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.023147 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.023887 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.024060 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:51.028149 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:51.028160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:51.028183 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:51.028233 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:51.032154 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:51.032923 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:51.032932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:51.032943 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:51.033169 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:51.033513 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:51.033518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:51.033521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:51.034281 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:51.034575 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:51.034888 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 03:01:51.035374 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":395,"threads_started":1}
I20260929 03:01:51.037277 9072 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } }
I20260929 03:01:51.037653 9072 raft_consensus.cc:374] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:51.037745 9072 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31052ca45b74cbc827b973e04cec020, State: Initialized, Role: FOLLOWER
I20260929 03:01:51.038303 9072 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } }
I20260929 03:01:51.038580 9072 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:01:51.038721 9072 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:01:51.038908 9072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:51.041584 9072 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } }
I20260929 03:01:51.042090 9072 leader_election.cc:302] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31052ca45b74cbc827b973e04cec020; no voters:
I20260929 03:01:51.042713 9072 leader_election.cc:288] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:01:51.042838 9077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:51.043769 9077 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 LEADER]: Becoming Leader. State: Replica: a31052ca45b74cbc827b973e04cec020, State: Running, Role: LEADER
I20260929 03:01:51.044281 9077 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } }
I20260929 03:01:51.044678 9072 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:51.046815 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31052ca45b74cbc827b973e04cec020. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } }
I20260929 03:01:51.046839 9078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } }
I20260929 03:01:51.047209 9078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:51.047209 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:51.048005 9084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:51.055459 9084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:51.065575 9084 catalog_manager.cc:1347] Generated new cluster ID: dc9d60ceb7a04bcc8109910b44efa848
I20260929 03:01:51.065661 9084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:51.089250 9084 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:01:51.091305 9084 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:51.103400 9084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Generated new TSK 0
I20260929 03:01:51.104579 9084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
I20260929 03:01:51.158464 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:51.492784 9098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:51.493126 9098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:51.493270 9098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:51.501236 9098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:51.501389 9098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:51.501547 9098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:51.512137 9098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:51.513584 9098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:51.514990 9098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:51.528023 9103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:51.528028 9104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:51.528239 9106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:51.529075 9098 server_base.cc:1034] running on GCE node
I20260929 03:01:51.529697 9098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:51.530680 9098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:51.531908 9098 hybrid_clock.cc:648] HybridClock initialized: now 1790650911531826 us; error 57 us; skew 500 ppm
I20260929 03:01:51.532234 9098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:51 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650911, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:51.535549 9098 init.cc:377] Logged in from keytab as oryx/127.2.205.129@KRBTEST.COM (short username oryx)
I20260929 03:01:51.537199 9098 webserver.cc:460] Webserver started at http://127.2.205.129:39143/ using document root <none> and password file <none>
I20260929 03:01:51.537912 9098 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:51.538019 9098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:51.538280 9098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:51.541456 9098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "954e63b28dce47778bc5ac4b7e19bc00"
format_stamp: "Formatted at 2026-09-29 03:01:51 on dist-test-slave-530n"
server_key: "b2507ce2419e45ce6b7709402f8e4130"
server_key_iv: "c3886930fc4ac64a89353f4c292af59e"
server_key_version: "encryptionkey@0"
I20260929 03:01:51.542248 9098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "954e63b28dce47778bc5ac4b7e19bc00"
format_stamp: "Formatted at 2026-09-29 03:01:51 on dist-test-slave-530n"
server_key: "b2507ce2419e45ce6b7709402f8e4130"
server_key_iv: "c3886930fc4ac64a89353f4c292af59e"
server_key_version: "encryptionkey@0"
I20260929 03:01:51.548194 9098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 03:01:51.553331 9113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:51.554972 9098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:01:51.555121 9098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "954e63b28dce47778bc5ac4b7e19bc00"
format_stamp: "Formatted at 2026-09-29 03:01:51 on dist-test-slave-530n"
server_key: "b2507ce2419e45ce6b7709402f8e4130"
server_key_iv: "c3886930fc4ac64a89353f4c292af59e"
server_key_version: "encryptionkey@0"
I20260929 03:01:51.555305 9098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:51.576021 9098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:51.581795 9098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:51.582118 9098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:51.583428 9098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:51.585165 9098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:51.585259 9098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:51.585390 9098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:51.585465 9098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:51.615815 9098 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37891
I20260929 03:01:51.615833 9225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37891 every 8 connection(s)
I20260929 03:01:51.617492 9098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:51.617805 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9098
I20260929 03:01:51.617986 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:01:51.618788 2870 external_mini_cluster.cc:1428] Setting key 987a56c86bb46fe4415d236a05a46b1a
Sep 29 03:01:51 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650911, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:51.637619 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:51.620250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:57157 (local address 127.2.205.190:40357)
0929 03:01:51.620708 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:51.620715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:51.621791 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 03:01:51.623586 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:51.623591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:51.623670 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:51.623764 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:51.625301 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.626019 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.627547 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.628126 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.628250 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:51.631380 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:51.631386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:51.631390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:51.631427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:51.634451 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:51.635155 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:51.635160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:51.635165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:51.635321 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:51.635703 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:51.635709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:51.635712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:51.636098 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:51.636257 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:51.636974 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 03:01:51.637238 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
WARNING: no policy specified for oryx/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:01:51.638741 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:51.620881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:57157)
0929 03:01:51.621641 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:51.621688 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:51.623320 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:51.624023 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:51.624033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:51.624525 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:51.625137 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:51.625156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.626167 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.626173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:51.627402 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:51.627414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:51.628325 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:51.628331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:51.628535 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:51.629135 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:51.629162 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:51.631178 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:51.634586 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:51.634593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:51.634606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:51.635020 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:51.635417 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:51.635421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:51.635425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:51.635594 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:51.636208 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:51.636214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:51.636575 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 03:01:51.636853 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 03:01:51.637446 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":293,"threads_started":1}
I20260929 03:01:51.640477 9226 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.130@KRBTEST.COM" created.
I20260929 03:01:51.640805 9226 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:51.641741 9226 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:51.644222 9037 ts_manager.cc:194] Registered new tserver with Master: 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891)
I20260929 03:01:51.645926 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:57157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
I20260929 03:01:51.664121 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:51.981379 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:51.981688 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:51.981828 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:51.989096 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:51.989234 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:51.989383 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:51.999248 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:52.000591 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:52.002027 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:52.014943 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:52.015000 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:52.015501 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:52.016081 9232 server_base.cc:1034] running on GCE node
I20260929 03:01:52.016697 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:52.017756 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:52.018953 9232 hybrid_clock.cc:648] HybridClock initialized: now 1790650912018888 us; error 64 us; skew 500 ppm
I20260929 03:01:52.019264 9232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:52 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650912, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:52.022547 9232 init.cc:377] Logged in from keytab as oryx/127.2.205.130@KRBTEST.COM (short username oryx)
I20260929 03:01:52.024165 9232 webserver.cc:460] Webserver started at http://127.2.205.130:41253/ using document root <none> and password file <none>
I20260929 03:01:52.024940 9232 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:52.025063 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:52.025341 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:52.028347 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "025271554e054a0a9118bfbaaa9c3005"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "01997bc13912a8401a818d1bf30c7357"
server_key_iv: "0cfdae4385783b52784b4c78357bce5f"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.029142 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "025271554e054a0a9118bfbaaa9c3005"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "01997bc13912a8401a818d1bf30c7357"
server_key_iv: "0cfdae4385783b52784b4c78357bce5f"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.034472 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:01:52.038128 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.039515 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:52.039678 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "025271554e054a0a9118bfbaaa9c3005"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "01997bc13912a8401a818d1bf30c7357"
server_key_iv: "0cfdae4385783b52784b4c78357bce5f"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.039875 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:52.050640 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:52.054396 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:52.054641 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:52.055512 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:52.056718 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:52.056819 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.056934 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:52.056985 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.086265 9232 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:36897
I20260929 03:01:52.086300 9359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:36897 every 8 connection(s)
I20260929 03:01:52.087546 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:52.093526 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9232
I20260929 03:01:52.093722 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:01:52.094161 2870 external_mini_cluster.cc:1428] Setting key 2bb351eb1338826a30aba731d926597d
Sep 29 03:01:52 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650912, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:52.109076 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:52.090121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:48333 (local address 127.2.205.190:40357)
0929 03:01:52.090279 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:52.090284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:52.091512 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 03:01:52.093579 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:52.093584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:52.093680 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:52.093796 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:52.095634 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.096427 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.098064 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.098471 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.098583 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:52.102520 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:52.102526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:52.102529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:52.102569 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:52.105515 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:52.106424 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:52.106429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:52.106431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:52.106579 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:52.107100 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:52.107105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:52.107107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:52.107429 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:52.107573 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:52.108427 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 03:01:52.108693 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:01:52.110169 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:52.090596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:48333)
0929 03:01:52.091324 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:52.091371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:52.093205 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:52.094084 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:52.094095 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:52.094697 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:52.095454 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:52.095475 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.096578 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.096582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:52.097912 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:52.097931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.098700 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.098707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:52.098942 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:52.099591 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:52.099615 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:52.102216 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:52.105719 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:52.105731 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:52.105750 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:52.106263 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:52.106712 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:52.106717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:52.106722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:52.106961 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:52.107576 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:52.107586 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:52.107998 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 03:01:52.108275 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 03:01:52.108962 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":278,"threads_started":1}
I20260929 03:01:52.111958 9360 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
I20260929 03:01:52.112279 9360 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:01:52.113104 9360 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:52.114703 9037 ts_manager.cc:194] Registered new tserver with Master: 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.131@KRBTEST.COM" created.
I20260929 03:01:52.115736 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:48333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
I20260929 03:01:52.140694 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:52.460842 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:52.461146 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:52.461282 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:52.468614 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:52.468735 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:52.468900 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:52.478849 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:52.480214 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:52.481637 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:52.495039 9370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:52.495086 9371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:52.495105 9373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:52.495851 9365 server_base.cc:1034] running on GCE node
I20260929 03:01:52.496496 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:52.497534 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:52.498755 9365 hybrid_clock.cc:648] HybridClock initialized: now 1790650912498710 us; error 31 us; skew 500 ppm
I20260929 03:01:52.499089 9365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:52 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650912, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:52.502243 9365 init.cc:377] Logged in from keytab as oryx/127.2.205.131@KRBTEST.COM (short username oryx)
I20260929 03:01:52.503666 9365 webserver.cc:460] Webserver started at http://127.2.205.131:46055/ using document root <none> and password file <none>
I20260929 03:01:52.504351 9365 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:52.504463 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:52.504738 9365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:52.507679 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "798ff083b26940d8b4f387dc23b14b1b"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "3b7d7b9cbab5ffa95797b77336c6ad0d"
server_key_iv: "93c0d7957e1a78fd55e6a134a5c60470"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.508421 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "798ff083b26940d8b4f387dc23b14b1b"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "3b7d7b9cbab5ffa95797b77336c6ad0d"
server_key_iv: "93c0d7957e1a78fd55e6a134a5c60470"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.513551 9365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:01:52.516968 9380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.518464 9365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:01:52.518625 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "798ff083b26940d8b4f387dc23b14b1b"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "3b7d7b9cbab5ffa95797b77336c6ad0d"
server_key_iv: "93c0d7957e1a78fd55e6a134a5c60470"
server_key_version: "encryptionkey@0"
I20260929 03:01:52.518821 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:52.540962 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:52.545496 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:52.545783 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:52.546752 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:52.548086 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:52.548164 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.548286 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:52.548339 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:52.577430 9365 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:39237
I20260929 03:01:52.577499 9492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:39237 every 8 connection(s)
I20260929 03:01:52.578805 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:52.579387 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9365
I20260929 03:01:52.579551 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:01:52.580108 2870 external_mini_cluster.cc:1428] Setting key 115751b6909fd5837dbd9d591cec8727
Sep 29 03:01:52 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650912, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:52.598807 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:52.581781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:56229 (local address 127.2.205.190:40357)
0929 03:01:52.581991 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:52.581997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:52.583800 (+ 1803us) server_negotiation.cc:407] Connection header received
0929 03:01:52.585958 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:52.585962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:52.586028 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:52.586108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:52.587540 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.588261 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.589853 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.590241 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.590330 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:52.593496 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:52.593501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:52.593504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:52.593536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:52.596333 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:52.597015 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:52.597018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:52.597021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:52.597142 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:52.597479 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:52.597482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:52.597483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:52.597729 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:52.597829 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:52.598414 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 03:01:52.598571 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:01:52.599743 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:52.582461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:56229)
0929 03:01:52.583512 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:52.583685 (+ 173us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:52.585718 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:52.586291 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:52.586299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:52.586823 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:52.587400 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:52.587418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.588368 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.588371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:52.589746 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:52.589759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:52.590404 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:52.590409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:52.590633 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:52.591185 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:52.591210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:52.593298 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:52.596444 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:52.596450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:52.596465 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:52.596914 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:52.597227 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:52.597230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:52.597232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:52.597398 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:52.597813 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:52.597818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:52.598120 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 03:01:52.598334 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 03:01:52.598703 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":325,"threads_started":1}
I20260929 03:01:52.601022 9493 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
I20260929 03:01:52.601287 9493 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:52.601939 9493 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:52.603416 9037 ts_manager.cc:194] Registered new tserver with Master: 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237)
I20260929 03:01:52.604173 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:56229
I20260929 03:01:52.607630 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.205.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.189@KRBTEST.COM" created.
I20260929 03:01:52.649037 9226 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.189.keytab.
Entry for principal oryx/127.2.205.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.189.keytab.
I20260929 03:01:52.655642 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:39387
--webserver_interface=127.2.205.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.189.keytab
--principal=oryx/127.2.205.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40357,127.2.205.189:39387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:01:52.980602 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:52.980947 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:52.981086 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:52.988372 9498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:52.988492 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:52.988569 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:52.988642 9498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:52.988703 9498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:52.998737 9498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40357,127.2.205.189:39387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.189.keytab
--principal=oryx/127.2.205.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.205.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
W20260929 03:01:52.999121 9498 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.205.190:40357,127.2.205.189:39387'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:01:53.000207 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:53.001670 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:53.014196 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:53.014180 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:53.014181 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:53.014909 9498 server_base.cc:1034] running on GCE node
I20260929 03:01:53.015666 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:53.017175 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:53.018417 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790650913018337 us; error 51 us; skew 500 ppm
I20260929 03:01:53.018761 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:53.021893 9498 init.cc:377] Logged in from keytab as oryx/127.2.205.189@KRBTEST.COM (short username oryx)
I20260929 03:01:53.023550 9498 webserver.cc:460] Webserver started at http://127.2.205.189:44727/ using document root <none> and password file <none>
I20260929 03:01:53.024284 9498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:53.024372 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:53.024644 9498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:53.027707 9498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/instance:
uuid: "7947e4e145c84b2499b48de3cf2671ed"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "7d33034a11c3b8202db20f1febf1634b"
server_key_iv: "88262c8769576e02cfceaa189e3d06aa"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.028524 9498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal/instance:
uuid: "7947e4e145c84b2499b48de3cf2671ed"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "7d33034a11c3b8202db20f1febf1634b"
server_key_iv: "88262c8769576e02cfceaa189e3d06aa"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.034247 9498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 03:01:53.037882 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:53.039342 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:01:53.039492 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "7947e4e145c84b2499b48de3cf2671ed"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "7d33034a11c3b8202db20f1febf1634b"
server_key_iv: "88262c8769576e02cfceaa189e3d06aa"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.039685 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:53.055209 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:53.058974 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:53.059186 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:53.076253 9498 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.189:39387
I20260929 03:01:53.076278 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.189:39387 every 8 connection(s)
I20260929 03:01:53.077641 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
I20260929 03:01:53.081477 9565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:53.084187 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9498
I20260929 03:01:53.091150 9565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.189@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
I20260929 03:01:53.111184 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.094943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.2.205.190:40357)
0929 03:01:53.095160 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.095171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.095875 (+ 704us) server_negotiation.cc:407] Connection header received
0929 03:01:53.096861 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.096865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.096937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.097048 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.098434 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.099186 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.100982 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.101527 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.101620 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.105503 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.105509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.105514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.105558 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.108129 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.108917 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.108922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.108926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.109067 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.109468 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.109471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.109473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.109779 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.109914 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.110641 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.110869 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:01:53.112417 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.086406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.2.205.189:39387)
0929 03:01:53.094097 (+ 7691us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.094112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.094162 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:01:53.096487 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.096519 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.096869 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.097042 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.098474 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.099861 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.101284 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.101964 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.102078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.106316 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.106327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.106345 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.106381 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.109105 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.109766 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.109773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.109775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.109893 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.110258 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.110261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.110263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.110615 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.110911 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.111238 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.111485 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":7436,"thread_start_us":7213,"threads_started":1}
I20260929 03:01:53.112447 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.094883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53426)
0929 03:01:53.095690 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.095748 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.096499 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.097129 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.097136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.097871 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.098246 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.098266 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.099340 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.099346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.100840 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.100852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.101740 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.101746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.102036 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.102894 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.102923 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.105290 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.108273 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.108286 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.108304 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.108794 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.109174 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.109180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.109184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.109358 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.109905 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.109913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.110266 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.110531 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.111071 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":3,"spinlock_wait_cycles":23936,"thread_start_us":295,"threads_started":1}
I20260929 03:01:53.114428 9565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:01:53.118564 9360 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.189@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
I20260929 03:01:53.127099 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.115025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51090)
0929 03:01:53.115178 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.115196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.115296 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.115660 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.115664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.116013 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.116144 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.116152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.117199 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.117203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.118061 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.118073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.118871 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.118876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.119097 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.119653 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.119669 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.121778 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.124864 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.124869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.124871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.125255 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.125671 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.125674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.125676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.125806 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.126282 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.126285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.126436 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.126552 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.126790 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 03:01:53.127166 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.115217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.2.205.189:39387)
0929 03:01:53.115353 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.115358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.115377 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:01:53.115461 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.115466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.115534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.115635 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.116268 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.117066 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.118190 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.118730 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.118859 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.121972 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.121978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.121981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.122013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.124694 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.125402 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.125407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.125410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.125565 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.125899 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.125903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.125905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.126191 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.126300 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.126681 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.126917 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":45}
I20260929 03:01:53.131876 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Bootstrap starting.
I20260929 03:01:53.135149 9565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:53.136286 9565 log.cc:824] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:53.139179 9565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: No bootstrap required, opened a new log
I20260929 03:01:53.146612 9565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } }
I20260929 03:01:53.147014 9565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:53.147132 9565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7947e4e145c84b2499b48de3cf2671ed, State: Initialized, Role: FOLLOWER
I20260929 03:01:53.147718 9565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } }
I20260929 03:01:53.148542 9571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } } }
I20260929 03:01:53.148887 9571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:53.149621 9565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:01:53.158169 9582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:01:53.158314 9582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:01:53.160502 9498 master_runner.cc:428] Detected that this master 7947e4e145c84b2499b48de3cf2671ed is joining an existing cluster
I20260929 03:01:53.160607 9498 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.205.189:39387
I20260929 03:01:53.179987 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.165574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.2.205.189:39387)
0929 03:01:53.165757 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.165763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.166266 (+ 503us) server_negotiation.cc:407] Connection header received
0929 03:01:53.166474 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.166479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.166544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.166618 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.168072 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.169105 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.171001 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.171626 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.171705 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.174056 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.174062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.174066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.174095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.177244 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.177991 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.177999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.178002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.178142 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.178500 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.178504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.178507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.178840 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.178972 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.179418 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.179638 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:53.181006 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.164529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.2.205.190:40357)
0929 03:01:53.164718 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.164725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.165343 (+ 618us) server_negotiation.cc:407] Connection header received
0929 03:01:53.165611 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.165616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.165693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.165779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.167594 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.168713 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.170887 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.171544 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.171641 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.174375 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.174381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.174386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.174428 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.177720 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.178544 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.178550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.178554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.178720 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.179188 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.179194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.179197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.179552 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.179674 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.180353 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.180608 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:01:53.181272 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.165356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51100)
0929 03:01:53.166175 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.166192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.166347 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.166795 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.166799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.167589 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.167920 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.167932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.169330 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.169335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.170873 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.170886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.171962 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.171966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.172170 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.172787 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.172806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.173899 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.177390 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.177395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.177399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.177876 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.178250 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.178254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.178257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.178407 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.179042 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.179046 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.179194 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.180552 (+ 1358us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.180874 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":101,"thread_start_us":547,"threads_started":1}
I20260929 03:01:53.182332 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.164549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53434)
0929 03:01:53.165236 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.165256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.165458 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.166255 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.166260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.166988 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.167417 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.167430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.168920 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.168926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.170722 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.170737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.171930 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.171936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.172235 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.172933 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.172955 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.174164 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.177880 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.177885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.177889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.178389 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.178867 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.178872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.178876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.179049 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.179767 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.179772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.179970 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.181503 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.181835 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":43,"spinlock_wait_cycles":33024,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for oryx/127.2.205.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.188@KRBTEST.COM" created.
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.190@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
I20260929 03:01:53.211081 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.189950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.2.205.189:39387)
0929 03:01:53.190142 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.190148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.193817 (+ 3669us) server_negotiation.cc:407] Connection header received
0929 03:01:53.194174 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.194183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.194304 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.194425 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.195860 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.197039 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.198779 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.199509 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.199656 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.203962 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.203970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.203975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.204019 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.207641 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.208697 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.208703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.208707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.208898 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.209331 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.209337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.209340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.209715 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.209868 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.210466 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.210712 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:01:53.211143 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.189939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51114)
0929 03:01:53.193638 (+ 3699us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.193686 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.193940 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.194507 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.194515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.195412 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.195680 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.195694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.197188 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.197193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.198614 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.198629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.199707 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.199714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.199930 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.200715 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.200741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.203707 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.207854 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.207870 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.207880 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.208551 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.209025 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.209030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.209034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.209210 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.209889 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.209918 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.210126 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.210345 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.210713 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3118,"thread_start_us":2980,"threads_started":1}
I20260929 03:01:53.213649 9037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.205.189:39387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed"
member_type: NON_VOTER
last_known_addr {
host: "127.2.205.189"
port: 39387
}
attrs {
promote: true
}
}
dest_uuid: "a31052ca45b74cbc827b973e04cec020"
cas_config_opid_index: -1
I20260929 03:01:53.214756 9037 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } }
I20260929 03:01:53.217710 9595 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.188.keytab.
Entry for principal oryx/127.2.205.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.188.keytab.
I20260929 03:01:53.218922 9540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:53.221730 9595 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.205.189:39387
I20260929 03:01:53.222118 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:35011
--webserver_interface=127.2.205.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.188.keytab
--principal=oryx/127.2.205.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:01:53.222496 9540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:01:53.222553 9498 master.cc:537] Master@127.2.205.189:39387 shutting down...
I20260929 03:01:53.223176 9595 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } }
I20260929 03:01:53.223368 9595 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:53.233556 9594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:01:53.237154 9024 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 03:01:53.239509 9498 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:53.240316 9498 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:53.241063 9498 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: stopping tablet replica
W20260929 03:01:53.247133 9024 consensus_peers.cc:487] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 -> Peer 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387): Couldn't send request to peer 7947e4e145c84b2499b48de3cf2671ed. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:01:53.251339 9498 master.cc:552] Master@127.2.205.189:39387 shutdown complete.
I20260929 03:01:53.251446 9498 master_runner.cc:305] Clearing existing system tablet
I20260929 03:01:53.251878 9498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:53.251996 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:53.254918 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 03:01:53.256541 9600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:53.256911 9498 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:53.257226 9498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:01:53.257349 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "7947e4e145c84b2499b48de3cf2671ed"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "7d33034a11c3b8202db20f1febf1634b"
server_key_iv: "88262c8769576e02cfceaa189e3d06aa"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.262646 9498 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:53.269635 9498 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:01:53.269901 9498 log.cc:1196] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:01:53.270437 9498 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Deleting consensus metadata
I20260929 03:01:53.270738 9498 master_runner.cc:315] Copying system tablet from 127.2.205.190:40357
I20260929 03:01:53.273300 9498 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.205.190:40357
I20260929 03:01:53.296319 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.274182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.2.205.190:40357)
0929 03:01:53.274372 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.274378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.281410 (+ 7032us) server_negotiation.cc:407] Connection header received
0929 03:01:53.281620 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.281627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.281734 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.281842 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.283028 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.284140 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.286073 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.286737 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.286890 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.289428 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.289435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.289440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.289485 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.293214 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.294075 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.294080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.294084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.294239 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.294655 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.294660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.294663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.295050 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.295205 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.295678 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.295952 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:01:53.296344 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.274126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53438)
0929 03:01:53.281241 (+ 7115us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.281269 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.281460 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.281886 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.281891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.282480 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.282790 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.282807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.284293 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.284299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.285887 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.285909 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.286911 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.286918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.287239 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.287980 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.288004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.289172 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.293385 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.293391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.293395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.293928 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.294361 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.294370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.294374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.294537 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.295165 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.295171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.295382 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.295527 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.295878 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6950,"thread_start_us":6787,"threads_started":1}
I20260929 03:01:53.297675 9057 tablet_copy_service.cc:140] P a31052ca45b74cbc827b973e04cec020: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7947e4e145c84b2499b48de3cf2671ed ({username='oryx', principal='oryx/127.2.205.189@KRBTEST.COM'} at 127.0.0.1:53438)
I20260929 03:01:53.297989 9057 tablet_copy_service.cc:161] P a31052ca45b74cbc827b973e04cec020: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7947e4e145c84b2499b48de3cf2671ed at {username='oryx', principal='oryx/127.2.205.189@KRBTEST.COM'} at 127.0.0.1:53438: session id = 7947e4e145c84b2499b48de3cf2671ed-00000000000000000000000000000000
I20260929 03:01:53.303015 9057 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:01:53.305727 9498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:01:53.313088 9498 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: tablet copy: Starting download of 0 data blocks...
I20260929 03:01:53.313468 9498 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: tablet copy: Starting download of 1 WAL segments...
I20260929 03:01:53.316418 9498 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:01:53.321019 9057 tablet_copy_service.cc:342] P a31052ca45b74cbc827b973e04cec020: Request end of tablet copy session 7947e4e145c84b2499b48de3cf2671ed-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.205.189@KRBTEST.COM'} at 127.0.0.1:53438
I20260929 03:01:53.321385 9057 tablet_copy_service.cc:433] P a31052ca45b74cbc827b973e04cec020: ending tablet copy session 7947e4e145c84b2499b48de3cf2671ed-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7947e4e145c84b2499b48de3cf2671ed
I20260929 03:01:53.333864 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:53.336791 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:53.342396 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:53.350114 9498 server_base.cc:1034] running on GCE node
W20260929 03:01:53.350859 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:53.351351 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:53.352025 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:53.353253 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790650913353181 us; error 34 us; skew 500 ppm
I20260929 03:01:53.353525 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:53.355836 9498 init.cc:377] Logged in from keytab as oryx/127.2.205.189@KRBTEST.COM (short username oryx)
I20260929 03:01:53.360790 9498 webserver.cc:460] Webserver started at http://127.2.205.189:37653/ using document root <none> and password file <none>
I20260929 03:01:53.361153 9498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:53.361230 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:53.364220 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:01:53.366184 9617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:53.366622 9498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:01:53.366796 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "7947e4e145c84b2499b48de3cf2671ed"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "7d33034a11c3b8202db20f1febf1634b"
server_key_iv: "88262c8769576e02cfceaa189e3d06aa"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.367014 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:53.405642 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:53.410600 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:53.433259 9498 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.189:39387
I20260929 03:01:53.433324 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.189:39387 every 8 connection(s)
I20260929 03:01:53.434353 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
I20260929 03:01:53.439862 9669 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:01:53.443821 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Bootstrap starting.
I20260929 03:01:53.463114 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:01:53.463660 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: Bootstrap complete.
I20260929 03:01:53.464804 9669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } }
I20260929 03:01:53.465090 9669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7947e4e145c84b2499b48de3cf2671ed, State: Initialized, Role: LEARNER
I20260929 03:01:53.465399 9669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } }
I20260929 03:01:53.466269 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } }
I20260929 03:01:53.466555 9669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:53.466678 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: LEARNER
I20260929 03:01:53.484104 9683 catalog_manager.cc:1259] Loaded cluster ID: dc9d60ceb7a04bcc8109910b44efa848
I20260929 03:01:53.484227 9683 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: loading cluster ID for follower catalog manager: success
I20260929 03:01:53.487949 9683 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: acquiring CA information for follower catalog manager: success
I20260929 03:01:53.490679 9683 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:01:53.606634 9493 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
W20260929 03:01:53.619892 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:53.620206 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:53.620292 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:53.627849 9598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:01:53.627974 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:53.628021 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:53.628074 9598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:01:53.628116 9598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:01:53.638597 9598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.188.keytab
--principal=oryx/127.2.205.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.205.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:53.639925 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:53.641247 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:53.653017 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:53.653261 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:53.653988 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:53.654260 9598 server_base.cc:1034] running on GCE node
I20260929 03:01:53.655064 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:53.656886 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:53.658098 9598 hybrid_clock.cc:648] HybridClock initialized: now 1790650913658050 us; error 35 us; skew 500 ppm
I20260929 03:01:53.658422 9598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:53.661345 9598 init.cc:377] Logged in from keytab as oryx/127.2.205.188@KRBTEST.COM (short username oryx)
I20260929 03:01:53.662734 9598 webserver.cc:460] Webserver started at http://127.2.205.188:33617/ using document root <none> and password file <none>
I20260929 03:01:53.663371 9598 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:53.663486 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:53.663774 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:01:53.666592 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/instance:
uuid: "e0e8bdcade5c42afa00caabed9279a14"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "b478e117e02d0360c98d5f168b5f76c4"
server_key_iv: "e8880f99e8214cd7d862478e7287e5e4"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.667321 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal/instance:
uuid: "e0e8bdcade5c42afa00caabed9279a14"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "b478e117e02d0360c98d5f168b5f76c4"
server_key_iv: "e8880f99e8214cd7d862478e7287e5e4"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.672088 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:01:53.675268 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:53.676527 9598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:01:53.676697 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "e0e8bdcade5c42afa00caabed9279a14"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "b478e117e02d0360c98d5f168b5f76c4"
server_key_iv: "e8880f99e8214cd7d862478e7287e5e4"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.676918 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:53.694299 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:53.698534 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:53.698807 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:53.718039 9598 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.188:35011
I20260929 03:01:53.718056 9749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.188:35011 every 8 connection(s)
I20260929 03:01:53.719527 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
I20260929 03:01:53.723712 9750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:53.725183 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9598
I20260929 03:01:53.735282 9750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.188@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:53.751179 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.727594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.2.205.188:35011)
0929 03:01:53.728461 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.728477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.728524 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:01:53.731182 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.731214 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.731741 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.732172 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.734304 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.735705 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.737655 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.738453 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.738661 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.743118 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.743133 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.743151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.743197 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.747164 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.747917 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.747924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.747937 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.748090 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.748435 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.748439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.748441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.749046 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.749315 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.749748 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.750156 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":383,"threads_started":1}
I20260929 03:01:53.753612 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.738768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.2.205.190:40357)
0929 03:01:53.738967 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.738971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.739691 (+ 720us) server_negotiation.cc:407] Connection header received
0929 03:01:53.739916 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.739921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.740000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.740081 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.741276 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.742043 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.743681 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.744237 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.744335 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.748301 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.748306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.748309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.748342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.750774 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.751484 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.751490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.751493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.751645 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.752095 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.752099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.752101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.752383 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.752548 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.753039 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.753275 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:01:53.753708 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.738507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53444)
0929 03:01:53.739534 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.739593 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.739761 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.740174 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.740181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.740866 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.741122 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.741131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.742176 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.742181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.743537 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.743551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.744422 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.744437 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.744658 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.745318 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.745342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.748135 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.750891 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.750902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.750910 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.751343 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.751786 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.751791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.751794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.751961 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.752531 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.752538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.752711 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.752931 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.753259 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":579,"threads_started":1}
I20260929 03:01:53.755390 9750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.188@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
I20260929 03:01:53.767298 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.755948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51116)
0929 03:01:53.756134 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.756152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.756260 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.756908 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.756911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.757185 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.757338 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.757346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.758572 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.758575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.759504 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.759514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.760186 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.760192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.760362 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.760813 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.760829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.762622 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.765108 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.765113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.765117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.765457 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.765788 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.765793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.765796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.765923 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.766358 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.766362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.766535 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.766692 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.766967 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 03:01:53.767530 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.756154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.2.205.189:39387)
0929 03:01:53.756625 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.756629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.756644 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:01:53.756687 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.756690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.756773 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.756861 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.757511 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.758442 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.759642 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.760046 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.760149 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.762790 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.762796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.762799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.762829 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.764999 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.765558 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.765564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.765566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.765691 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.766014 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.766019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.766022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.766255 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.766884 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.766996 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.767191 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":254,"threads_started":1}
I20260929 03:01:53.768420 9750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650913, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.188@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
I20260929 03:01:53.778899 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.769136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.0.0.1:56910)
0929 03:01:53.769310 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.769327 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.769412 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.769611 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.769614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.769888 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.770008 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.770014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.770824 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.770827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.771606 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.771616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.772206 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.772209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.772338 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.772774 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.772796 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.774524 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.776854 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.776857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.776860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.777175 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.777448 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.777451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.777453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.777583 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.778001 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.778005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.778139 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.778252 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.778492 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:01:53.778949 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.769102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.2.205.188:35011)
0929 03:01:53.769237 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.769241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.769366 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:01:53.769476 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.769479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.769525 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.769590 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.770109 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.770734 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.771707 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.772082 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.772167 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.774675 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.774679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.774682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.774707 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.776727 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.777257 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.777261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.777263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.777362 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.777660 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.777663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.777665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.777908 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.778010 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.778345 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.778625 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":44}
I20260929 03:01:53.783705 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Bootstrap starting.
I20260929 03:01:53.787030 9750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:53.788592 9750 log.cc:824] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:53.791438 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: No bootstrap required, opened a new log
I20260929 03:01:53.798769 9750 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } }
I20260929 03:01:53.799170 9750 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:53.799297 9750 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e8bdcade5c42afa00caabed9279a14, State: Initialized, Role: FOLLOWER
I20260929 03:01:53.799886 9750 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } }
I20260929 03:01:53.800791 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } } }
I20260929 03:01:53.801101 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:53.801826 9750 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:01:53.810708 9768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:01:53.810858 9768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:01:53.815495 9598 master_runner.cc:428] Detected that this master e0e8bdcade5c42afa00caabed9279a14 is joining an existing cluster
I20260929 03:01:53.815618 9598 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.205.188:35011
I20260929 03:01:53.822409 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.814655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.2.205.189:39387)
0929 03:01:53.814843 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.814850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.815271 (+ 421us) server_negotiation.cc:407] Connection header received
0929 03:01:53.815449 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.815454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.815542 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.815628 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:53.816159 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.817073 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.819856 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.821173 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.821309 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.821548 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.822033 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.822169 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:01:53.822432 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.814471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51128)
0929 03:01:53.815189 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.815208 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.815333 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.815798 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.815801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.815829 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:53.816029 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.816038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.817182 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.817187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.818916 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.818933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.821661 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.821666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.821796 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.821843 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.821929 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.822067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":497,"threads_started":1}
I20260929 03:01:53.823910 9644 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 03:01:53.824469 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31052ca45b74cbc827b973e04cec020. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } }
I20260929 03:01:53.824687 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: LEARNER
I20260929 03:01:53.849465 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.831496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.2.205.190:40357)
0929 03:01:53.832209 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.832215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.832627 (+ 412us) server_negotiation.cc:407] Connection header received
0929 03:01:53.832900 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.832907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.832997 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.833082 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.835121 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.836236 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.838437 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.839075 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.839176 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.841978 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.841985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.841990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.842034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.846248 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847101 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.847107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.847111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.847287 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847758 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.847763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.847766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.848106 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.848232 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.848854 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.849082 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260929 03:01:53.849821 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.831779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.2.205.188:35011)
0929 03:01:53.832027 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.832033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.833254 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 03:01:53.833474 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.833478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.833547 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.833616 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.836227 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.837455 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.839643 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.840354 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.840432 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.843381 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.843387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.843392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.843435 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.846876 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847610 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.847615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.847618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.847762 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.848161 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.848166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.848169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.848590 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.848722 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.849244 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.849452 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":70}
I20260929 03:01:53.850926 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.833327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.2.205.189:39387)
0929 03:01:53.833495 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.833500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.834914 (+ 1414us) server_negotiation.cc:407] Connection header received
0929 03:01:53.835045 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.835049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.835123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.835208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.836494 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.837543 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.839660 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.840308 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.840406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.844349 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.844356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.844361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.844400 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.847585 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.848323 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.848328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.848331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.848472 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.848909 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.848913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.848916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.849242 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.849351 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.849911 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.850148 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:01:53.851047 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.831704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.0.0.1:56922)
0929 03:01:53.833174 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.833191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.833348 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.833818 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.833823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.835708 (+ 1885us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.836060 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.836075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.837673 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.837678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.839489 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.839506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.840698 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.840703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.840945 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.841588 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.841612 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.843204 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.847020 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.847028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.847485 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.847891 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.847898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.848059 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.848830 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.848834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.849006 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.850053 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.850336 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"mutex_wait_us":87,"spinlock_wait_cycles":50688,"thread_start_us":370,"threads_started":1}
I20260929 03:01:53.851743 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.831704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53456)
0929 03:01:53.832498 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.832519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.832689 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.833307 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.833312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.834109 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.834934 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.834948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.836421 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.836437 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.838237 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.838255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.839463 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.839469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.839719 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.840451 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.840476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.841727 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.846407 (+ 4680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.846412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.846416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.846954 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.847431 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.847439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.847635 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.848307 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.848311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.848566 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.850271 (+ 1705us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.850602 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":73,"spinlock_wait_cycles":68608,"thread_start_us":392,"threads_started":1}
I20260929 03:01:53.851804 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.833627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51140)
0929 03:01:53.834805 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.834819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.834939 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.835374 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.835379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.836012 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.836329 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.836340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.837696 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.837700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.839528 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.839537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.840687 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.840693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.840900 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.841682 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.841700 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.844131 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.847720 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.847728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.847731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.848194 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.848610 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.848614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.848617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.848797 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.849442 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.849446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.849649 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.850683 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.851132 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":30,"spinlock_wait_cycles":5376,"thread_start_us":556,"threads_started":1}
I20260929 03:01:53.857290 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9098
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.190@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
Sep 29 03:01:53 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.129@KRBTEST.COM
I20260929 03:01:53.873121 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.856716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.0.0.1:56928)
0929 03:01:53.857010 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.857029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.857139 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.857525 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.857531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.857987 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.858166 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.858177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.859632 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.859638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.861038 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.861054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.862078 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.862084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.862254 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.862841 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.862864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.866140 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.870081 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.870086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.870090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.870577 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.871042 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.871048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.871052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.871422 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.872124 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.872129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.872306 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.872438 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.872717 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260929 03:01:53.873479 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.856787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.2.205.188:35011)
0929 03:01:53.857039 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.857044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.857080 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:01:53.857283 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.857288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.857393 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.857495 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.858372 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.859487 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.861224 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.861900 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.866297 (+ 4397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.866488 (+ 191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.866494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.866502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.866531 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.869946 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.870705 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.870711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.870715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.870854 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.871555 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.871560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.871567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.871916 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.872060 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.872948 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.873145 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 03:01:53.875180 9037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.205.188:35011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14"
member_type: NON_VOTER
last_known_addr {
host: "127.2.205.188"
port: 35011
}
attrs {
promote: true
}
}
dest_uuid: "a31052ca45b74cbc827b973e04cec020"
cas_config_opid_index: 5
I20260929 03:01:53.876211 9037 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } }
I20260929 03:01:53.876484 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.858175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.2.205.129:37891)
0929 03:01:53.861217 (+ 3042us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.861224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.861248 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:01:53.861298 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.861303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.861497 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.861634 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.862865 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.863831 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.865343 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.866136 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.866251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.870810 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.870822 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.870837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.870867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.873733 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.874436 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.874445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.874449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.874595 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.874981 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.874986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.874988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.875291 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.875565 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.875876 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.876130 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2889,"thread_start_us":2727,"threads_started":1}
I20260929 03:01:53.877975 9221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:53.878350 9770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.881848 9644 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:01:53.883481 9725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:53.888832 9780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:01:53.890102 9770 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.205.188:35011
I20260929 03:01:53.890790 9725 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:01:53.894368 9780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.894711 9780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:53.895392 9644 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.893762 9598 master.cc:537] Master@127.2.205.188:35011 shutting down...
I20260929 03:01:53.902400 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: NON_VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: true } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.902704 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: LEARNER
I20260929 03:01:53.903802 9780 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: attempting to promote NON_VOTER 7947e4e145c84b2499b48de3cf2671ed to VOTER
I20260929 03:01:53.904000 9781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:53.906908 9780 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } }
I20260929 03:01:53.909389 9644 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 LEARNER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 03:01:53.910107 9781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 03:01:53.913169 9780 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.915632 9643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
W20260929 03:01:53.916266 9022 consensus_peers.cc:487] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 -> Peer e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188:35011): Couldn't send request to peer e0e8bdcade5c42afa00caabed9279a14. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:01:53.917423 9780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.917656 9780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:53.917879 9781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.918085 9781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:53.919327 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:53.919531 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:53.937628 9757 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:53.937945 9757 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } }
I20260929 03:01:53.942135 9757 leader_election.cc:288] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387)
I20260929 03:01:53.943353 9643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7947e4e145c84b2499b48de3cf2671ed" is_pre_election: true
I20260929 03:01:53.943322 9047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a31052ca45b74cbc827b973e04cec020" is_pre_election: true
I20260929 03:01:53.944399 9702 leader_election.cc:302] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0e8bdcade5c42afa00caabed9279a14; no voters: 7947e4e145c84b2499b48de3cf2671ed, a31052ca45b74cbc827b973e04cec020
I20260929 03:01:53.944949 9598 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:53.945221 9757 raft_consensus.cc:2684] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260929 03:01:53.945571 9598 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:53.945763 9598 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: stopping tablet replica
I20260929 03:01:53.952081 9598 master.cc:552] Master@127.2.205.188:35011 shutdown complete.
I20260929 03:01:53.952171 9598 master_runner.cc:305] Clearing existing system tablet
I20260929 03:01:53.952538 9598 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:53.952632 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:53.955075 9598 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 03:01:53.956383 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:53.956681 9598 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:53.956945 9598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:01:53.957043 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "e0e8bdcade5c42afa00caabed9279a14"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "b478e117e02d0360c98d5f168b5f76c4"
server_key_iv: "e8880f99e8214cd7d862478e7287e5e4"
server_key_version: "encryptionkey@0"
I20260929 03:01:53.960246 9598 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:53.964706 9598 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:01:53.964864 9598 log.cc:1196] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:01:53.965245 9598 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Deleting consensus metadata
I20260929 03:01:53.965451 9598 master_runner.cc:315] Copying system tablet from 127.2.205.190:40357
I20260929 03:01:53.967746 9598 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: tablet copy: Beginning tablet copy session from remote peer at address 127.2.205.190:40357
I20260929 03:01:53.979991 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.968850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.2.205.190:40357)
0929 03:01:53.969076 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:53.969082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:53.969186 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:01:53.969397 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:53.969403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:53.969476 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:53.969557 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:53.970362 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.971091 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.972313 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.972733 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.972879 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.974941 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:53.974945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:53.974949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:53.974988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:53.977481 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.978083 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.978087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.978090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.978212 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:53.978601 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:53.978606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:53.978610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:53.978888 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:53.979030 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:53.979408 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 03:01:53.979619 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:01:53.980031 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:53.968396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53458)
0929 03:01:53.968900 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:53.969108 (+ 208us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:53.969274 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:53.969599 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:53.969604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:53.970011 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:53.970218 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.970226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.971226 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.971232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.972174 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:53.972186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:53.972885 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:53.972891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:53.973075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:53.974047 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:53.974064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:53.974762 (+ 698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:53.977604 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.977609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.977614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.977970 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.978339 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:53.978343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:53.978345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:53.978497 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:53.979014 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:53.979018 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:53.979171 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 03:01:53.979287 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:01:53.979591 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":590,"threads_started":1}
I20260929 03:01:53.981017 9057 tablet_copy_service.cc:140] P a31052ca45b74cbc827b973e04cec020: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0e8bdcade5c42afa00caabed9279a14 ({username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:53458)
I20260929 03:01:53.981177 9057 tablet_copy_service.cc:161] P a31052ca45b74cbc827b973e04cec020: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0e8bdcade5c42afa00caabed9279a14 at {username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:53458: session id = e0e8bdcade5c42afa00caabed9279a14-00000000000000000000000000000000
I20260929 03:01:53.983426 9057 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:01:53.985226 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:01:53.990087 9598 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: tablet copy: Starting download of 0 data blocks...
I20260929 03:01:53.990377 9598 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: tablet copy: Starting download of 1 WAL segments...
I20260929 03:01:53.992656 9598 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:01:53.996206 9057 tablet_copy_service.cc:342] P a31052ca45b74cbc827b973e04cec020: Request end of tablet copy session e0e8bdcade5c42afa00caabed9279a14-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:53458
I20260929 03:01:53.996783 9057 tablet_copy_service.cc:433] P a31052ca45b74cbc827b973e04cec020: ending tablet copy session e0e8bdcade5c42afa00caabed9279a14-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0e8bdcade5c42afa00caabed9279a14
I20260929 03:01:54.019552 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:54.022119 9794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:54.027324 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9098
W20260929 03:01:54.030079 9795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:54.047413 9598 server_base.cc:1034] running on GCE node
W20260929 03:01:54.048578 9797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:54.048810 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:37891
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:40357
--webserver_port=39143
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:01:54.053097 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:54.060079 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:54.061271 9598 hybrid_clock.cc:648] HybridClock initialized: now 1790650914061232 us; error 32 us; skew 500 ppm
I20260929 03:01:54.061503 9598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650914, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:54.063271 9598 init.cc:377] Logged in from keytab as oryx/127.2.205.188@KRBTEST.COM (short username oryx)
I20260929 03:01:54.064553 9598 webserver.cc:460] Webserver started at http://127.2.205.188:44147/ using document root <none> and password file <none>
I20260929 03:01:54.064855 9598 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:54.064952 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:54.067437 9598 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:01:54.069187 9805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:54.069557 9598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:01:54.069672 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "e0e8bdcade5c42afa00caabed9279a14"
format_stamp: "Formatted at 2026-09-29 03:01:53 on dist-test-slave-530n"
server_key: "b478e117e02d0360c98d5f168b5f76c4"
server_key_iv: "e8880f99e8214cd7d862478e7287e5e4"
server_key_version: "encryptionkey@0"
I20260929 03:01:54.069835 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:54.086256 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:54.090301 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:54.113844 9598 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.188:35011
I20260929 03:01:54.113855 9856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.188:35011 every 8 connection(s)
I20260929 03:01:54.114809 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
I20260929 03:01:54.120832 9857 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:01:54.125397 9857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Bootstrap starting.
I20260929 03:01:54.142046 9857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:01:54.142514 9857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Bootstrap complete.
I20260929 03:01:54.143507 9857 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } }
I20260929 03:01:54.143826 9857 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0e8bdcade5c42afa00caabed9279a14, State: Initialized, Role: LEARNER
I20260929 03:01:54.144243 9857 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } }
I20260929 03:01:54.146243 9860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:54.146616 9857 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:01:54.146782 9860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:01:54.165215 9871 catalog_manager.cc:1259] Loaded cluster ID: dc9d60ceb7a04bcc8109910b44efa848
I20260929 03:01:54.165324 9871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: loading cluster ID for follower catalog manager: success
I20260929 03:01:54.168059 9871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: acquiring CA information for follower catalog manager: success
I20260929 03:01:54.169988 9871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:01:54.348037 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.341682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.0.0.1:56942)
0929 03:01:54.342002 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:54.342025 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:54.342184 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:54.343096 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:54.343101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:54.343130 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:54.343283 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.343291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.344664 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.344669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.345992 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.346006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.347274 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.347279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.347490 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.347526 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:01:54.347638 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:01:54.347787 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260929 03:01:54.348410 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.341845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.2.205.188:35011)
0929 03:01:54.342731 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:54.342737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:54.342762 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:01:54.342845 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:54.342850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:54.342934 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:54.343059 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:54.343472 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.344519 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.346185 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.347073 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.347242 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.347495 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:54.347690 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:01:54.347849 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":572,"threads_started":1}
I20260929 03:01:54.349234 9832 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 03:01:54.349778 9860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31052ca45b74cbc827b973e04cec020. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:54.350056 9860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: LEARNER
W20260929 03:01:54.456724 9799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:54.457064 9799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:54.457197 9799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:54.464303 9799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:54.464414 9799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:54.464555 9799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:01:54.474224 9799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:37891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=39143
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:54.475471 9799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:54.477115 9799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:54.491034 9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:54.491052 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:54.491586 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:54.492281 9799 server_base.cc:1034] running on GCE node
I20260929 03:01:54.492978 9799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:54.493966 9799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:54.495185 9799 hybrid_clock.cc:648] HybridClock initialized: now 1790650914495133 us; error 42 us; skew 500 ppm
I20260929 03:01:54.495518 9799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650914, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:54.499012 9799 init.cc:377] Logged in from keytab as oryx/127.2.205.129@KRBTEST.COM (short username oryx)
I20260929 03:01:54.500562 9799 webserver.cc:460] Webserver started at http://127.2.205.129:39143/ using document root <none> and password file <none>
I20260929 03:01:54.501298 9799 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:54.501428 9799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:54.507018 9799 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:01:54.509929 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:54.511327 9799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:01:54.511489 9799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "954e63b28dce47778bc5ac4b7e19bc00"
format_stamp: "Formatted at 2026-09-29 03:01:51 on dist-test-slave-530n"
server_key: "b2507ce2419e45ce6b7709402f8e4130"
server_key_iv: "c3886930fc4ac64a89353f4c292af59e"
server_key_version: "encryptionkey@0"
I20260929 03:01:54.512526 9799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:54.529438 9799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:54.534209 9799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:54.534474 9799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:54.535475 9799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:54.536998 9799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:54.537096 9799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:54.537189 9799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:54.537256 9799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:54.569331 9799 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37891
I20260929 03:01:54.569383 9999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37891 every 8 connection(s)
I20260929 03:01:54.570772 9799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:01:54.573452 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 9799
I20260929 03:01:54.573815 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9232
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650914, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650914, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.130@KRBTEST.COM
Sep 29 03:01:54 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650914, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:54.613847 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.575412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.2.205.130:36897)
0929 03:01:54.588051 (+ 12639us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:54.588063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:54.589231 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 03:01:54.590521 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:54.590531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:54.590792 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:54.590975 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:54.593068 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.594765 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.597328 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.598255 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.598383 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.605912 (+ 7529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:54.605939 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:54.605961 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:54.606002 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:54.609631 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.610586 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.610596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.610599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.610767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.611226 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.611232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.611235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.611694 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:54.611996 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:54.613166 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0929 03:01:54.613439 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12380,"thread_start_us":12671,"threads_started":1}
I20260929 03:01:54.614600 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.573792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:34231 (local address 127.2.205.188:35011)
0929 03:01:54.573982 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:54.573987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:54.589710 (+ 15723us) server_negotiation.cc:407] Connection header received
0929 03:01:54.592223 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:54.592229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:54.592316 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:54.592384 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:54.597584 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.598645 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.600541 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.601175 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.601269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.605885 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:54.605892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:54.605897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:54.605941 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:54.609169 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.611619 (+ 2450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.611624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.611628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.611783 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612343 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.612348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.612352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.612703 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:54.612863 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:54.614050 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0929 03:01:54.614285 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:01:54.614616 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.590407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:58491 (local address 127.2.205.190:40357)
0929 03:01:54.590846 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:54.590852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:54.597433 (+ 6581us) server_negotiation.cc:407] Connection header received
0929 03:01:54.597700 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:54.597706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:54.597793 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:54.597872 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:54.599461 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.600598 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.602653 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.603298 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.603405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.607215 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:54.607221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:54.607226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:54.607270 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:54.611380 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612236 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.612242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.612245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.612390 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612838 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.612842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.612846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.613258 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:54.613424 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:54.614042 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 03:01:54.614268 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":222,"threads_started":1}
I20260929 03:01:54.615454 9355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:54.616462 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.574625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:54621 (local address 127.2.205.189:39387)
0929 03:01:54.575367 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:54.575373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:54.589881 (+ 14508us) server_negotiation.cc:407] Connection header received
0929 03:01:54.592314 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:54.592319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:54.592396 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:54.592485 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:54.595378 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.596344 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.599309 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.599995 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.600095 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.607081 (+ 6986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:54.607089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:54.607095 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:54.607144 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:54.611778 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612573 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.612579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.612582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.612730 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:54.613694 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:54.613699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:54.613702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:54.614052 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:54.614171 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:54.615012 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 03:01:54.615917 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":519,"threads_started":1}
I20260929 03:01:54.617075 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.574500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.2.205.129:34231)
0929 03:01:54.589488 (+ 14988us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:54.589561 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:54.591872 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:54.592912 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:54.592925 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:54.593943 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:54.597379 (+ 3436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.597396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.598868 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.598873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.600348 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.600360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.601548 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.601554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.601723 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.602258 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:54.602295 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:54.605616 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:54.609586 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.609596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.609616 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.611476 (+ 1860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.611926 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.611931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.611934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.612211 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.613092 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:54.613101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:54.613605 (+ 504us) client_negotiation.cc:769] Sending connection context
0929 03:01:54.615009 (+ 1404us) client_negotiation.cc:240] Negotiation successful
0929 03:01:54.615633 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14598,"mutex_wait_us":2590,"spinlock_wait_cycles":6656,"thread_start_us":14367,"threads_started":1}
I20260929 03:01:54.617098 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.590872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:58491)
0929 03:01:54.597307 (+ 6435us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:54.597335 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:54.597555 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:54.598133 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:54.598138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:54.598814 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:54.599256 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.599269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.600826 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.600831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.602470 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.602485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.603655 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.603659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.603781 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.604271 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:54.604287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:54.606970 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:54.611541 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.611546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.611550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.612091 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.612526 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.612534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.612690 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.613498 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:54.613503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:54.613648 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 03:01:54.614864 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0929 03:01:54.615493 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6288,"mutex_wait_us":30,"thread_start_us":6111,"threads_started":1}
I20260929 03:01:54.618969 10001 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
I20260929 03:01:54.619328 10001 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:54.620272 10001 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:54.621192 10000 heartbeater.cc:347] Connected to a master server at 127.2.205.188:35011
I20260929 03:01:54.621330 10000 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:54.621661 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:54.575545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.2.205.129:54621)
0929 03:01:54.589771 (+ 14226us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:54.589788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:54.591883 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:54.592912 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:54.592925 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:54.593943 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:54.595135 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.595168 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.596580 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.596586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.599095 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:54.599129 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:54.600478 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:54.600486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:54.600793 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:54.601630 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:54.601666 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:54.606366 (+ 4700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:54.611930 (+ 5564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.611935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.611939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.612442 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.612892 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:54.612897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:54.612900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:54.613571 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:54.614246 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:54.614250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:54.614388 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 03:01:54.620813 (+ 6425us) client_negotiation.cc:240] Negotiation successful
0929 03:01:54.621144 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14090,"mutex_wait_us":71,"spinlock_wait_cycles":60800,"thread_start_us":13890,"threads_started":1}
I20260929 03:01:54.621805 10000 heartbeater.cc:511] Master 127.2.205.188:35011 requested a full tablet report, sending...
I20260929 03:01:54.622335 9037 ts_manager.cc:194] Re-registered known tserver with Master: 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891)
I20260929 03:01:54.622762 10002 heartbeater.cc:347] Connected to a master server at 127.2.205.189:39387
I20260929 03:01:54.622887 10002 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:54.623260 10002 heartbeater.cc:511] Master 127.2.205.189:39387 requested a full tablet report, sending...
I20260929 03:01:54.623485 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:58491
I20260929 03:01:54.623699 9822 ts_manager.cc:194] Registered new tserver with Master: 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891)
I20260929 03:01:54.624886 9634 ts_manager.cc:194] Registered new tserver with Master: 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891)
I20260929 03:01:54.702528 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9232
I20260929 03:01:54.715822 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:36897
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:40357
--webserver_port=41253
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:01:54.889496 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: NON_VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: true } } }
I20260929 03:01:54.889847 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:54.891377 10015 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: attempting to promote NON_VOTER e0e8bdcade5c42afa00caabed9279a14 to VOTER
I20260929 03:01:54.892262 10015 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:54.894513 9832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 LEARNER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 03:01:54.894588 9643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Refusing update from remote peer a31052ca45b74cbc827b973e04cec020: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 03:01:54.895087 10013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:01:54.895471 10014 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 03:01:54.898234 10015 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.899889 9832 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.902730 9643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.908469 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.909847 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:54.910321 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.910701 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:54.911089 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.911422 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:54.911511 10019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.910316 10015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a31052ca45b74cbc827b973e04cec020" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:54.911777 10019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:54.911866 10015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020 [sys.catalog]: This master's current role is: LEADER
W20260929 03:01:55.042869 10012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:55.043143 10012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:55.043232 10012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:55.050539 10012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:55.050642 10012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:55.050757 10012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:01:55.060416 10012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:36897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=41253
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:55.061801 10012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:55.063264 10012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:55.076960 10029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:55.076910 10026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:55.076929 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:55.077757 10012 server_base.cc:1034] running on GCE node
I20260929 03:01:55.078379 10012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:55.079481 10012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:55.080679 10012 hybrid_clock.cc:648] HybridClock initialized: now 1790650915080626 us; error 49 us; skew 500 ppm
I20260929 03:01:55.081071 10012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:55.084112 10012 init.cc:377] Logged in from keytab as oryx/127.2.205.130@KRBTEST.COM (short username oryx)
I20260929 03:01:55.085552 10012 webserver.cc:460] Webserver started at http://127.2.205.130:41253/ using document root <none> and password file <none>
I20260929 03:01:55.086217 10012 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:55.086357 10012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:55.091560 10012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:01:55.094208 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:55.095506 10012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:01:55.095630 10012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "025271554e054a0a9118bfbaaa9c3005"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "01997bc13912a8401a818d1bf30c7357"
server_key_iv: "0cfdae4385783b52784b4c78357bce5f"
server_key_version: "encryptionkey@0"
I20260929 03:01:55.096536 10012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:55.130246 10012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:55.134804 10012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:55.135092 10012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:55.136135 10012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:55.137616 10012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:55.137699 10012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:55.137796 10012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:55.137866 10012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:55.168286 10012 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:36897
I20260929 03:01:55.168366 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:36897 every 8 connection(s)
I20260929 03:01:55.169926 10012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:01:55.175992 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10012
I20260929 03:01:55.176311 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9365
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.131@KRBTEST.COM
I20260929 03:01:55.208636 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.180576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:45945 (local address 127.2.205.190:40357)
0929 03:01:55.189279 (+ 8703us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.189285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.189305 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:55.189375 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.189380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.189453 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.189566 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.191708 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.192701 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.195270 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.195920 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.196050 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.201576 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.201581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.201585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.201637 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.205432 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.206251 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.206257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.206261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.206431 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.206914 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.206919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.206922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.207249 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.207366 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.208027 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.208253 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8582,"thread_start_us":8459,"threads_started":1}
I20260929 03:01:55.209544 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.174522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.2.205.130:34703)
0929 03:01:55.180097 (+ 5575us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.180118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.180233 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.189005 (+ 8772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.189021 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.190128 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.191796 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.191806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.193198 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.193205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.195127 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.195142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.197294 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.197300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.197862 (+ 562us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.198361 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.198377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.202808 (+ 4431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.206647 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.206651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.206655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.207207 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.207628 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.207634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.207637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.207768 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.208399 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.208404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.208549 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.208683 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.208987 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"mutex_wait_us":1067,"thread_start_us":5294,"threads_started":1}
I20260929 03:01:55.209609 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.173954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.2.205.130:48715)
0929 03:01:55.176350 (+ 2396us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.176399 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.178871 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.189604 (+ 10733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.189610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.190124 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.191134 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.191165 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.192644 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.192650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.195098 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.195124 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.195928 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.195935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.196165 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.197125 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.197157 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.200017 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.204398 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.204407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.204425 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.205037 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.205479 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.205484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.205487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.205767 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.206522 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.206530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.207088 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.207411 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.208008 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2082,"mutex_wait_us":172,"spinlock_wait_cycles":14592,"thread_start_us":1846,"threads_started":1}
I20260929 03:01:55.210222 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.172985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:48715 (local address 127.2.205.189:39387)
0929 03:01:55.189099 (+ 16114us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.189105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.189124 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:01:55.189182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.189188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.189268 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.189388 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.191357 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.192463 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.195296 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.195690 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.195814 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.200293 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.200301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.200307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.200357 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.204244 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.205186 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.205192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.205196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.205342 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.205897 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.205902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.205905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.206284 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.206434 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.209632 (+ 3198us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.209886 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15994,"thread_start_us":15846,"threads_started":1}
I20260929 03:01:55.210916 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.172990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:34703 (local address 127.2.205.188:35011)
0929 03:01:55.188265 (+ 15275us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.188271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.188297 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:01:55.188366 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.188372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.188453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.188582 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.191968 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.193009 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.196270 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.196942 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.197058 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.203117 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.203124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.203129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.203172 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.206511 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.207350 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.207355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.207358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.207504 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.207879 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.207884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.207887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.208212 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.208331 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.210321 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.210558 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15132,"thread_start_us":15815,"threads_started":1}
I20260929 03:01:55.210925 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.180409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:45945)
0929 03:01:55.185981 (+ 5572us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.185998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.186091 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.190745 (+ 4654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.190749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.191226 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.191498 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.191509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.193182 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.193187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.195113 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.195124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.196556 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.196562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.196723 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.198561 (+ 1838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.198584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.201354 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.205617 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.205622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.205626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.206102 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.206563 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.206568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.206572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.206778 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.207445 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.207449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.207592 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.207749 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.210570 (+ 2821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5452,"mutex_wait_us":2625,"thread_start_us":5302,"threads_started":1}
I20260929 03:01:55.211575 10151 heartbeater.cc:347] Connected to a master server at 127.2.205.189:39387
I20260929 03:01:55.211935 10151 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.212229 10150 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
I20260929 03:01:55.212374 10150 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.213465 10149 heartbeater.cc:347] Connected to a master server at 127.2.205.188:35011
I20260929 03:01:55.213538 10151 heartbeater.cc:511] Master 127.2.205.189:39387 requested a full tablet report, sending...
I20260929 03:01:55.213618 10149 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.213748 10150 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:55.214073 10149 heartbeater.cc:511] Master 127.2.205.188:35011 requested a full tablet report, sending...
I20260929 03:01:55.214246 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.177424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.2.205.131:39237)
0929 03:01:55.190153 (+ 12729us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.190162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.190192 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:01:55.190259 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.190266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.190537 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.190697 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.192662 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.194743 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.196583 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.197533 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.197659 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.204345 (+ 6686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.204366 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.204386 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.204441 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.208107 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.209453 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.209463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.209467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.210099 (+ 632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.210778 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.210783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.210786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.211534 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.212221 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.213309 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.213668 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12553,"thread_start_us":12391,"threads_started":1}
I20260929 03:01:55.215467 9822 ts_manager.cc:194] Registered new tserver with Master: 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:55.215737 9488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:55.216427 9634 ts_manager.cc:194] Registered new tserver with Master: 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:55.216569 9037 ts_manager.cc:194] Re-registered known tserver with Master: 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:55.217592 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:45945
I20260929 03:01:55.314256 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9365
I20260929 03:01:55.327342 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:39237
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:40357
--webserver_port=46055
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:01:55.626909 10001 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
W20260929 03:01:55.654655 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:01:55.654942 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:01:55.655079 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:01:55.662052 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:01:55.662140 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:01:55.662253 10162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:01:55.671520 10162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=46055
--tserver_master_addrs=127.2.205.190:40357,127.2.205.189:39387,127.2.205.188:35011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:01:55.672667 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:01:55.673893 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:01:55.686414 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:55.686339 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:01:55.686367 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:01:55.687434 10162 server_base.cc:1034] running on GCE node
I20260929 03:01:55.688081 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:01:55.689102 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:01:55.690299 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790650915690246 us; error 41 us; skew 500 ppm
I20260929 03:01:55.690614 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:01:55.693702 10162 init.cc:377] Logged in from keytab as oryx/127.2.205.131@KRBTEST.COM (short username oryx)
I20260929 03:01:55.695191 10162 webserver.cc:460] Webserver started at http://127.2.205.131:46055/ using document root <none> and password file <none>
I20260929 03:01:55.695891 10162 fs_manager.cc:359] Metadata directory not provided
I20260929 03:01:55.696003 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:01:55.701683 10162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:01:55.704597 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:55.706063 10162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:01:55.706212 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "798ff083b26940d8b4f387dc23b14b1b"
format_stamp: "Formatted at 2026-09-29 03:01:52 on dist-test-slave-530n"
server_key: "3b7d7b9cbab5ffa95797b77336c6ad0d"
server_key_iv: "93c0d7957e1a78fd55e6a134a5c60470"
server_key_version: "encryptionkey@0"
I20260929 03:01:55.707243 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:01:55.733791 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:01:55.737918 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:01:55.738189 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:01:55.739159 10162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:01:55.740546 10162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:01:55.740634 10162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:01:55.740736 10162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:01:55.740825 10162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 03:01:55.770239 10162 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:39237
I20260929 03:01:55.770334 10289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:39237 every 8 connection(s)
I20260929 03:01:55.771687 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:01:55.776588 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10162
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.188@KRBTEST.COM
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:01:55.838382 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.795399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.2.205.188:35011)
0929 03:01:55.808935 (+ 13536us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.808941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.811031 (+ 2090us) server_negotiation.cc:407] Connection header received
0929 03:01:55.811313 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.811319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.811403 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.811480 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.815543 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.816870 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.822271 (+ 5401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.822959 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.823053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.831547 (+ 8494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.831555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.831560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.831598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.835098 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.835901 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.835907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.835910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.836079 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.836516 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.836520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.836523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.836908 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.837115 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.837748 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.837984 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13255,"thread_start_us":13360,"threads_started":1}
I20260929 03:01:55.839934 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.774584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:37735 (local address 127.2.205.188:35011)
0929 03:01:55.790100 (+ 15516us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.790107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.811304 (+ 21197us) server_negotiation.cc:407] Connection header received
0929 03:01:55.811917 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.811923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.812007 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.812084 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.815087 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.816462 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.823357 (+ 6895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.823944 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.824057 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.830413 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.830420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.830424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.830464 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.835986 (+ 5522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.837039 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.837043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.837047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.837173 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.837729 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.837734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.837737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.838063 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.838188 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.839304 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.839542 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15346,"thread_start_us":15532,"threads_started":1}
I20260929 03:01:55.842479 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.776146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.2.205.131:37735)
0929 03:01:55.811219 (+ 35073us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.811238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.811750 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.812704 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.812773 (+ 69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.813802 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.814835 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.814869 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.816776 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.816782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.819087 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.819103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.824391 (+ 5288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.824399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.824686 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.825652 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.825684 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.830139 (+ 4455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.836193 (+ 6054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.836204 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.836225 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.836908 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.837325 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.837329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.837333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.837613 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.838285 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.838294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.838794 (+ 500us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.840208 (+ 1414us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.840851 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34951,"thread_start_us":34030,"threads_started":1}
Sep 29 03:01:55 dist-test-slave-530n krb5kdc[8992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650915, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.189@KRBTEST.COM
I20260929 03:01:55.844239 10290 heartbeater.cc:347] Connected to a master server at 127.2.205.188:35011
I20260929 03:01:55.844532 10290 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.845863 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.809044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.2.205.189:39387)
0929 03:01:55.809627 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.809633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.811133 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 03:01:55.811403 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.811410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.811500 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.811583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.819479 (+ 7896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.822069 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.827269 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.827905 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.828007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.834534 (+ 6527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.834540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.834545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.834599 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.841953 (+ 7354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.842855 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.842861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.842865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.843040 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.843850 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.843854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.843857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.844247 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.844470 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.845121 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.845398 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":314,"threads_started":1}
I20260929 03:01:55.846153 10290 heartbeater.cc:511] Master 127.2.205.188:35011 requested a full tablet report, sending...
I20260929 03:01:55.848207 9822 ts_manager.cc:194] Registered new tserver with Master: 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237)
I20260929 03:01:55.848779 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.774573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:57249 (local address 127.2.205.190:40357)
0929 03:01:55.804888 (+ 30315us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.804895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.807214 (+ 2319us) server_negotiation.cc:407] Connection header received
0929 03:01:55.810113 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.810119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.810217 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.810324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.820100 (+ 9776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.821252 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.823251 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.823842 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.823961 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.837858 (+ 13897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.837865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.837869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.837912 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.842159 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.845200 (+ 3041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.845206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.845210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.845378 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.846197 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.846203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.846206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.846583 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.846732 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.848188 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.848395 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30140,"thread_start_us":48543,"threads_started":1}
I20260929 03:01:55.848826 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.775555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:57249)
0929 03:01:55.806964 (+ 31409us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.807045 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.809669 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.816172 (+ 6503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.816178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.819671 (+ 3493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.819922 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.819934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.821479 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.821485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.822882 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.822896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.827283 (+ 4387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.827294 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.827482 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.834606 (+ 7124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.834625 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.837633 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.844075 (+ 6442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.844081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.844085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.845062 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.845860 (+ 798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.845865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.845869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.846079 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.847759 (+ 1680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.847764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.847919 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.848046 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.848362 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30951,"thread_start_us":39323,"threads_started":1}
I20260929 03:01:55.850332 10291 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40357
I20260929 03:01:55.850459 10291 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.850867 10291 heartbeater.cc:511] Master 127.2.205.190:40357 requested a full tablet report, sending...
I20260929 03:01:55.851814 9037 ts_manager.cc:194] Re-registered known tserver with Master: 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237)
I20260929 03:01:55.852794 9037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:57249
I20260929 03:01:55.853740 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.795446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.2.205.190:40357)
0929 03:01:55.825123 (+ 29677us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.825128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.825146 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:55.825194 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.825198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.825272 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.825381 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.830212 (+ 4831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.831384 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.833578 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.834214 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.834331 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.842424 (+ 8093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.842431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.842436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.842477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.847744 (+ 5267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.849568 (+ 1824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.849574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.849577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.849716 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.850291 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.850295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.850299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.850639 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.850890 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.851282 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.853283 (+ 2001us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27781,"thread_start_us":9158,"threads_started":1}
I20260929 03:01:55.855360 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.816858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:42129 (local address 127.2.205.189:39387)
0929 03:01:55.823677 (+ 6819us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.823681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.823696 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:01:55.823737 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.823740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.823793 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.823885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:55.825129 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.831384 (+ 6255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.837822 (+ 6438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.838459 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.838590 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.846861 (+ 8271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:55.846868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:55.846873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:55.846929 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:55.851800 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.852421 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.852427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.852430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.852577 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:55.852989 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:55.852994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:55.852997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:55.853372 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:55.853503 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.854801 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.855026 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6717,"thread_start_us":6595,"threads_started":1}
I20260929 03:01:55.855608 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.815518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.2.205.131:42129)
0929 03:01:55.821041 (+ 5523us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.821060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.821166 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.823960 (+ 2794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.823966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.824565 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:01:55.824947 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.824963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.832943 (+ 7980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.832949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.834391 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.834406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.838894 (+ 4488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.838901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.839130 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.841823 (+ 2693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:01:55.841851 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:01:55.846607 (+ 4756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:01:55.851915 (+ 5308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.851921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.851932 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.852278 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.852682 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:01:55.852685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:01:55.852688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:01:55.852866 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:01:55.853610 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:01:55.853615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:01:55.853849 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.853994 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.854291 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5392,"thread_start_us":5650,"threads_started":1}
I20260929 03:01:55.856387 10292 heartbeater.cc:347] Connected to a master server at 127.2.205.189:39387
I20260929 03:01:55.856516 10292 heartbeater.cc:464] Registering TS with master...
I20260929 03:01:55.856925 10292 heartbeater.cc:511] Master 127.2.205.189:39387 requested a full tablet report, sending...
I20260929 03:01:55.858026 9634 ts_manager.cc:194] Registered new tserver with Master: 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237)
I20260929 03:01:55.858242 9037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:55.865243 9037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:55.893427 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.882245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.2.205.131:39237)
0929 03:01:55.884657 (+ 2412us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.884663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.884690 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:01:55.884805 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.884813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.885118 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.885341 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:55.886021 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.888320 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.890802 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.892286 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.892396 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.892580 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.892936 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.893104 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":2149,"threads_started":1}
I20260929 03:01:55.893474 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.881923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:39237 (local address 127.0.0.1:46502)
0929 03:01:55.882528 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.882548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.882676 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.885336 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.885342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.885375 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:55.885816 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.885831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.888490 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.888494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.890678 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.890689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.892521 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.892527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.892701 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.892741 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.892862 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.893013 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":288,"threads_started":1}
I20260929 03:01:55.897143 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.885218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:36897 (local address 127.0.0.1:52024)
0929 03:01:55.888891 (+ 3673us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.888908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.889030 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.889783 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.889787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.889813 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:55.890596 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.890607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.892371 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.892377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.894592 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.894609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.896299 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.896306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.896485 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.896520 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.896585 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.896705 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3276,"mutex_wait_us":28,"thread_start_us":547,"threads_started":1}
I20260929 03:01:55.898118 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.882360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37891 (local address 127.0.0.1:57004)
0929 03:01:55.888886 (+ 6526us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:55.888907 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:55.889052 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:55.889963 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:55.889969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:55.889991 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:55.890434 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.890446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.894294 (+ 3848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.894299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.895749 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:55.895761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.897038 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.897045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:55.897229 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.897262 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:01:55.897341 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:01:55.897459 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"mutex_wait_us":64,"thread_start_us":2488,"threads_started":1}
I20260929 03:01:55.897696 10225 tablet_service.cc:1467] Processing CreateTablet for tablet db88d23e1f024318a64124f0ebf7710e (DEFAULT_TABLE table=test-table [id=f1973669580f46228d0840eacc6dfe2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:55.898629 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.885495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.2.205.130:36897)
0929 03:01:55.889280 (+ 3785us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.889288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.889313 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:01:55.889392 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.889399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.889569 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.889761 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:55.890787 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.892242 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.894747 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.896089 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.897492 (+ 1403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.897990 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.898065 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.898375 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3616,"thread_start_us":3229,"threads_started":1}
I20260929 03:01:55.898679 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:55.882728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.2.205.129:37891)
0929 03:01:55.883254 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:55.883264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:55.889016 (+ 5752us) server_negotiation.cc:407] Connection header received
0929 03:01:55.889202 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:55.889210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:55.889511 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:55.889717 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:55.890605 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.894181 (+ 3576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.895899 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:55.896843 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:55.896973 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:55.897182 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:55.898326 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0929 03:01:55.898480 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":977,"threads_started":1}
I20260929 03:01:55.900200 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88d23e1f024318a64124f0ebf7710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:55.902204 9935 tablet_service.cc:1467] Processing CreateTablet for tablet db88d23e1f024318a64124f0ebf7710e (DEFAULT_TABLE table=test-table [id=f1973669580f46228d0840eacc6dfe2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:55.902205 10084 tablet_service.cc:1467] Processing CreateTablet for tablet db88d23e1f024318a64124f0ebf7710e (DEFAULT_TABLE table=test-table [id=f1973669580f46228d0840eacc6dfe2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:55.903618 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88d23e1f024318a64124f0ebf7710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:55.903635 9935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db88d23e1f024318a64124f0ebf7710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:55.912783 10324 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Bootstrap starting.
I20260929 03:01:55.916710 10324 tablet_bootstrap.cc:654] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:55.918355 10324 log.cc:824] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:55.918649 10325 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Bootstrap starting.
I20260929 03:01:55.919606 10326 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Bootstrap starting.
I20260929 03:01:55.922710 10326 tablet_bootstrap.cc:654] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:55.923692 10325 tablet_bootstrap.cc:654] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:55.924022 10326 log.cc:824] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:55.924780 10325 log.cc:824] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Log is configured to *not* fsync() on all Append() calls
I20260929 03:01:55.926069 10324 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: No bootstrap required, opened a new log
I20260929 03:01:55.926398 10324 ts_tablet_manager.cc:1397] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260929 03:01:55.927304 10326 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: No bootstrap required, opened a new log
I20260929 03:01:55.927606 10326 ts_tablet_manager.cc:1397] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 03:01:55.928576 10325 tablet_bootstrap.cc:492] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: No bootstrap required, opened a new log
I20260929 03:01:55.928891 10325 ts_tablet_manager.cc:1397] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 03:01:55.934712 10324 raft_consensus.cc:348] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.935071 10324 raft_consensus.cc:374] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:55.935163 10324 raft_consensus.cc:729] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798ff083b26940d8b4f387dc23b14b1b, State: Initialized, Role: FOLLOWER
I20260929 03:01:55.935721 10324 consensus_queue.cc:260] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.937569 10326 raft_consensus.cc:348] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.937969 10326 raft_consensus.cc:374] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:55.938047 10326 raft_consensus.cc:729] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025271554e054a0a9118bfbaaa9c3005, State: Initialized, Role: FOLLOWER
I20260929 03:01:55.938644 10324 ts_tablet_manager.cc:1428] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 03:01:55.938683 10326 consensus_queue.cc:260] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.938879 10291 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
I20260929 03:01:55.939111 10325 raft_consensus.cc:348] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.939487 10325 raft_consensus.cc:374] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:55.939574 10325 raft_consensus.cc:729] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954e63b28dce47778bc5ac4b7e19bc00, State: Initialized, Role: FOLLOWER
I20260929 03:01:55.940181 10325 consensus_queue.cc:260] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:55.944785 10325 ts_tablet_manager.cc:1428] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260929 03:01:55.945220 10326 ts_tablet_manager.cc:1428] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 03:01:55.946014 10150 heartbeater.cc:503] Master 127.2.205.190:40357 was elected leader, sending a full tablet report...
I20260929 03:01:56.022820 10331 raft_consensus.cc:482] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:56.023154 10331 raft_consensus.cc:504] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:56.024874 10331 leader_election.cc:288] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237), 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:56.031792 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.025334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:51819 (local address 127.2.205.130:36897)
0929 03:01:56.025695 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.025701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.026243 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:01:56.026386 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.026390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.026450 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.026523 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:56.027542 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.028247 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.030030 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.030809 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.030972 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.031050 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.031400 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.031528 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260929 03:01:56.031788 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.025275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:48813 (local address 127.2.205.131:39237)
0929 03:01:56.025511 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.025519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.026033 (+ 514us) server_negotiation.cc:407] Connection header received
0929 03:01:56.026230 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.026233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.026292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.026368 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:56.027419 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.028152 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.030033 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.030800 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.030970 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.031049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.031400 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.031528 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:01:56.031934 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.025094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:39237 (local address 127.2.205.129:48813)
0929 03:01:56.025913 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:56.025933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:56.026088 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:56.026396 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:56.026401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:56.026425 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:56.027244 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:56.027259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.028281 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.028286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:56.029871 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:56.029883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.031034 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.031040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:56.031229 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.031270 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:01:56.031360 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:01:56.031520 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":61,"thread_start_us":790,"threads_started":1}
I20260929 03:01:56.032208 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.025211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:36897 (local address 127.2.205.129:51819)
0929 03:01:56.026137 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:56.026151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:56.026272 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:56.026566 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:56.026571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:56.026597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:56.027367 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:56.027377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.028383 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.028388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:56.029870 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:56.029882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.031085 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.031091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:56.031230 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.031274 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:01:56.031353 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:01:56.031525 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":66,"thread_start_us":384,"threads_started":1}
I20260929 03:01:56.033008 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db88d23e1f024318a64124f0ebf7710e" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798ff083b26940d8b4f387dc23b14b1b" is_pre_election: true
I20260929 03:01:56.033509 10245 raft_consensus.cc:2454] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 0.
I20260929 03:01:56.033411 10104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db88d23e1f024318a64124f0ebf7710e" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025271554e054a0a9118bfbaaa9c3005" is_pre_election: true
I20260929 03:01:56.033989 10104 raft_consensus.cc:2454] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 0.
I20260929 03:01:56.034176 9889 leader_election.cc:302] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 798ff083b26940d8b4f387dc23b14b1b, 954e63b28dce47778bc5ac4b7e19bc00; no voters:
I20260929 03:01:56.034667 10331 raft_consensus.cc:2781] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:56.034895 10331 raft_consensus.cc:482] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:56.035025 10331 raft_consensus.cc:3037] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:56.037627 10331 raft_consensus.cc:504] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:56.038187 10331 leader_election.cc:288] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 election: Requested vote from peers 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237), 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:56.038770 10104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db88d23e1f024318a64124f0ebf7710e" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025271554e054a0a9118bfbaaa9c3005"
I20260929 03:01:56.038753 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db88d23e1f024318a64124f0ebf7710e" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798ff083b26940d8b4f387dc23b14b1b"
I20260929 03:01:56.038965 10245 raft_consensus.cc:3037] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:56.038976 10104 raft_consensus.cc:3037] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:56.041251 10245 raft_consensus.cc:2454] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 1.
I20260929 03:01:56.041263 10104 raft_consensus.cc:2454] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 1.
W20260929 03:01:56.041662 10293 tablet.cc:2369] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:56.041697 9890 leader_election.cc:302] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025271554e054a0a9118bfbaaa9c3005, 954e63b28dce47778bc5ac4b7e19bc00; no voters:
I20260929 03:01:56.042001 10331 raft_consensus.cc:2781] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:56.042478 10331 raft_consensus.cc:686] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 LEADER]: Becoming Leader. State: Replica: 954e63b28dce47778bc5ac4b7e19bc00, State: Running, Role: LEADER
I20260929 03:01:56.042994 10331 consensus_queue.cc:237] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:56.047952 9037 catalog_manager.cc:5495] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "954e63b28dce47778bc5ac4b7e19bc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } health_report { overall_health: UNKNOWN } } }
W20260929 03:01:56.074541 10003 tablet.cc:2369] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:01:56.102476 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.096914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.2.205.129:37891)
0929 03:01:56.097151 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.097158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.097244 (+ 86us) server_negotiation.cc:407] Connection header received
0929 03:01:56.097478 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.097482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.097563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.097705 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:56.098097 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.098840 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.100158 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.100591 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.100702 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.101237 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:56.101834 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:56.101988 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.102081 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.102190 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:01:56.111464 10245 raft_consensus.cc:1261] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Refusing update from remote peer 954e63b28dce47778bc5ac4b7e19bc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:56.111450 10104 raft_consensus.cc:1261] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Refusing update from remote peer 954e63b28dce47778bc5ac4b7e19bc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:56.112686 10335 consensus_queue.cc:1035] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:56.113107 10331 consensus_queue.cc:1035] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:56.129119 10338 mvcc.cc:205] Tried to move back new op lower bound from 7334506152381038592 to 7334506152115818496. Current Snapshot: MvccSnapshot[applied={T|T < 7334506152381038592 or (T in {7334506152381038592})}]
I20260929 03:01:56.145433 9036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:56.145821 9036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:01:56.149675 10340 mvcc.cc:205] Tried to move back new op lower bound from 7334506152381038592 to 7334506152115818496. Current Snapshot: MvccSnapshot[applied={T|T < 7334506152381038592 or (T in {7334506152381038592})}]
I20260929 03:01:56.154397 9036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a31052ca45b74cbc827b973e04cec020: Sending DeleteTablet for 3 replicas of tablet db88d23e1f024318a64124f0ebf7710e
I20260929 03:01:56.155455 10225 tablet_service.cc:1514] Processing DeleteTablet for tablet db88d23e1f024318a64124f0ebf7710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:56 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46502
I20260929 03:01:56.156138 9935 tablet_service.cc:1514] Processing DeleteTablet for tablet db88d23e1f024318a64124f0ebf7710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:56 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:57004
I20260929 03:01:56.156497 10084 tablet_service.cc:1514] Processing DeleteTablet for tablet db88d23e1f024318a64124f0ebf7710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:01:56 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:52024
I20260929 03:01:56.156900 10348 tablet_replica.cc:330] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: stopping tablet replica
I20260929 03:01:56.157371 10348 raft_consensus.cc:2229] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:01:56.157972 10348 raft_consensus.cc:2258] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:56.160061 10348 ts_tablet_manager.cc:1905] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:56.168097 10348 ts_tablet_manager.cc:1918] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:56.168260 10348 log.cc:1196] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/db88d23e1f024318a64124f0ebf7710e
I20260929 03:01:56.168875 10348 ts_tablet_manager.cc:1939] T db88d23e1f024318a64124f0ebf7710e P 954e63b28dce47778bc5ac4b7e19bc00: Deleting consensus metadata
I20260929 03:01:56.170543 9023 catalog_manager.cc:4841] TS 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891): tablet db88d23e1f024318a64124f0ebf7710e (table test-table [id=f1973669580f46228d0840eacc6dfe2a]) successfully deleted
I20260929 03:01:56.172034 10349 tablet_replica.cc:330] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: stopping tablet replica
I20260929 03:01:56.172456 10349 raft_consensus.cc:2229] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:56.172967 10349 raft_consensus.cc:2258] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:56.175128 10349 ts_tablet_manager.cc:1905] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:56.177790 10347 tablet_replica.cc:330] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: stopping tablet replica
I20260929 03:01:56.179352 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9005
I20260929 03:01:56.179934 10347 raft_consensus.cc:2229] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:01:56.180438 9067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:01:56.182717 10349 ts_tablet_manager.cc:1918] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:56.182905 10349 log.cc:1196] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/db88d23e1f024318a64124f0ebf7710e
I20260929 03:01:56.183842 10349 ts_tablet_manager.cc:1939] T db88d23e1f024318a64124f0ebf7710e P 025271554e054a0a9118bfbaaa9c3005: Deleting consensus metadata
I20260929 03:01:56.186499 9023 catalog_manager.cc:4841] TS 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897): tablet db88d23e1f024318a64124f0ebf7710e (table test-table [id=f1973669580f46228d0840eacc6dfe2a]) successfully deleted
I20260929 03:01:56.186929 10347 raft_consensus.cc:2258] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:01:56.191123 10347 ts_tablet_manager.cc:1905] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:01:56.199764 10347 ts_tablet_manager.cc:1918] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:01:56.200102 10347 log.cc:1196] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/db88d23e1f024318a64124f0ebf7710e
I20260929 03:01:56.200871 10347 ts_tablet_manager.cc:1939] T db88d23e1f024318a64124f0ebf7710e P 798ff083b26940d8b4f387dc23b14b1b: Deleting consensus metadata
I20260929 03:01:56.209859 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.159499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.2.205.189:39387)
0929 03:01:56.159661 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.159666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.177092 (+ 17426us) server_negotiation.cc:407] Connection header received
0929 03:01:56.177296 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.177302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.177404 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.177508 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:56.190027 (+ 12519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.191260 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.193717 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.194175 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.194298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.202469 (+ 8171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:56.202477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:56.202481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:56.202531 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:56.206032 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.206952 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.206959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.206963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.207120 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.207553 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.207559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.207563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.207944 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:56.208180 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.208547 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.208807 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:01:56.226800 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.178044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.2.205.188:35011)
0929 03:01:56.178254 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.178260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.209177 (+ 30917us) server_negotiation.cc:407] Connection header received
0929 03:01:56.209356 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.209362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.209464 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.209564 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:56.211700 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.213484 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.215563 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.216278 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.216554 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.219917 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:56.219924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:56.219929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:56.219977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:56.223622 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.224556 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.224562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.224566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.224725 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.225157 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.225162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.225166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.225540 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:56.225786 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.226199 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.226450 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 03:01:56.317111 9022 catalog_manager.cc:4841] TS 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237): tablet db88d23e1f024318a64124f0ebf7710e (table test-table [id=f1973669580f46228d0840eacc6dfe2a]) successfully deleted
I20260929 03:01:56.318698 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9005
I20260929 03:01:56.365029 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.350391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.2.205.189:39387)
0929 03:01:56.350552 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.350557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.351234 (+ 677us) server_negotiation.cc:407] Connection header received
0929 03:01:56.351377 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.351383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.351458 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.351546 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:56.353424 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.354343 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.356176 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.356782 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.356894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.359803 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:56.359808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:56.359811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:56.359841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:56.362253 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.362841 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.362846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.362850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.363002 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.363394 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.363399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.363402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.363691 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:56.363930 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.364280 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.364490 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:01:56.365752 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:56.350124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.2.205.188:35011)
0929 03:01:56.350286 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:56.350292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:56.350493 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:01:56.350764 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:56.350769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:56.350855 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:56.350928 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:01:56.352920 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.353919 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.355397 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:56.355825 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:56.355949 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:56.359455 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:01:56.359462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:01:56.359466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:01:56.359505 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:01:56.362817 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.363479 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.363485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.363488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.363622 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:01:56.364013 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:01:56.364019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:01:56.364022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:01:56.364329 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:01:56.364548 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:56.365056 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 03:01:56.365270 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:01:57.142347 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.141634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:56141)
0929 03:01:57.142042 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.142171 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":213,"threads_started":1}
W20260929 03:01:57.142740 10150 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:40357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.157137 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.156355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:46347)
0929 03:01:57.156815 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.156961 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":225,"threads_started":1}
W20260929 03:01:57.157509 10291 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:40357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.158289 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.157699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:32833)
0929 03:01:57.158053 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.158151 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":172,"threads_started":1}
W20260929 03:01:57.158618 10001 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:40357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.654735 10361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a31052ca45b74cbc827b973e04cec020)
I20260929 03:01:57.655035 10362 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a31052ca45b74cbc827b973e04cec020)
I20260929 03:01:57.655061 10361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:57.655248 10362 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:57.656032 10362 leader_election.cc:288] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387)
I20260929 03:01:57.656628 10361 leader_election.cc:288] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188:35011)
I20260929 03:01:57.657547 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.656261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53962)
0929 03:01:57.657043 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.657255 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":285,"threads_started":1}
I20260929 03:01:57.657740 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.656791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53978)
0929 03:01:57.657271 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.657493 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":432,"threads_started":1}
W20260929 03:01:57.658553 9621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
W20260929 03:01:57.664098 9809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.666419 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.657120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:35011 (local address 127.0.0.1:56984)
0929 03:01:57.657936 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.657955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:57.658091 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:57.659133 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:57.659138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:57.659166 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:57.659448 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:57.659460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.661338 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.661344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:57.663621 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:57.663630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.665429 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.665435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:57.665718 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:57.665761 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:01:57.665880 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:01:57.666053 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":48,"threads_started":1}
I20260929 03:01:57.666450 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.657382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.2.205.188:35011)
0929 03:01:57.658805 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:57.658812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:57.658831 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:01:57.658898 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:57.658903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:57.658975 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:57.659590 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:57.659813 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.661144 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.663773 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.665207 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.665615 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:57.665728 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:57.665909 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:01:57.666063 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":945,"threads_started":1}
I20260929 03:01:57.668509 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.657516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.2.205.189:39387)
0929 03:01:57.658567 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:57.658572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:57.658591 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:01:57.658659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:57.658665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:57.658789 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:57.659382 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:57.659640 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.660963 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.663549 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.667285 (+ 3736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.667422 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:57.667545 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:57.668015 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 03:01:57.668148 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":925,"thread_start_us":754,"threads_started":1}
I20260929 03:01:57.668907 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.656942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:39387 (local address 127.0.0.1:51174)
0929 03:01:57.657915 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.657935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:57.658091 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:57.658971 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:57.658978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:57.659009 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:57.659244 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:57.659255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.661132 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.661138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:57.663315 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:57.663333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:57.667478 (+ 4145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:57.667485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:57.667761 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:57.667830 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 03:01:57.667938 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:01:57.668118 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":6823,"threads_started":1}
I20260929 03:01:57.669508 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.668931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53994)
0929 03:01:57.669131 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.669297 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:01:57.669811 9809 leader_election.cc:334] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.670127 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.669719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:53996)
0929 03:01:57.669887 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.669994 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:01:57.670367 9621 leader_election.cc:334] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.670413 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7947e4e145c84b2499b48de3cf2671ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0e8bdcade5c42afa00caabed9279a14" is_pre_election: true
I20260929 03:01:57.670758 9832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7947e4e145c84b2499b48de3cf2671ed in term 1.
I20260929 03:01:57.670869 9643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7947e4e145c84b2499b48de3cf2671ed" is_pre_election: true
I20260929 03:01:57.671118 9643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e8bdcade5c42afa00caabed9279a14 in term 1.
I20260929 03:01:57.671289 9619 leader_election.cc:302] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7947e4e145c84b2499b48de3cf2671ed, e0e8bdcade5c42afa00caabed9279a14; no voters: a31052ca45b74cbc827b973e04cec020
I20260929 03:01:57.671528 9809 leader_election.cc:302] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7947e4e145c84b2499b48de3cf2671ed, e0e8bdcade5c42afa00caabed9279a14; no voters: a31052ca45b74cbc827b973e04cec020
I20260929 03:01:57.671679 10361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:01:57.671804 10361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader a31052ca45b74cbc827b973e04cec020)
I20260929 03:01:57.671895 10361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:01:57.672886 10362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:01:57.673015 10362 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a31052ca45b74cbc827b973e04cec020)
I20260929 03:01:57.673107 10362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:01:57.674752 10361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:57.675515 10361 leader_election.cc:288] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 election: Requested vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), e0e8bdcade5c42afa00caabed9279a14 (127.2.205.188:35011)
I20260929 03:01:57.676040 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.675642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54000)
0929 03:01:57.675790 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.675895 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:01:57.675961 10362 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:57.676324 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7947e4e145c84b2499b48de3cf2671ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0e8bdcade5c42afa00caabed9279a14"
I20260929 03:01:57.676585 9832 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7947e4e145c84b2499b48de3cf2671ed in current term 2: Already voted for candidate e0e8bdcade5c42afa00caabed9279a14 in this term.
I20260929 03:01:57.676582 10362 leader_election.cc:288] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 election: Requested vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387)
I20260929 03:01:57.677124 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.676701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54008)
0929 03:01:57.676942 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.677021 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 03:01:57.677796 9643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7947e4e145c84b2499b48de3cf2671ed"
I20260929 03:01:57.678079 9643 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e0e8bdcade5c42afa00caabed9279a14 in current term 2: Already voted for candidate 7947e4e145c84b2499b48de3cf2671ed in this term.
I20260929 03:01:57.678185 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.677828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54014)
0929 03:01:57.678000 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.678072 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 03:01:57.678264 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:57.677839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54030)
0929 03:01:57.678011 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:57.678114 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:01:57.678424 9621 leader_election.cc:334] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
W20260929 03:01:57.678434 9809 leader_election.cc:334] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:57.678627 9621 leader_election.cc:302] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7947e4e145c84b2499b48de3cf2671ed; no voters: a31052ca45b74cbc827b973e04cec020, e0e8bdcade5c42afa00caabed9279a14
I20260929 03:01:57.678714 9809 leader_election.cc:302] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0e8bdcade5c42afa00caabed9279a14; no voters: 7947e4e145c84b2499b48de3cf2671ed, a31052ca45b74cbc827b973e04cec020
I20260929 03:01:57.678903 10361 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:01:57.678944 10362 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:01:58.144344 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:58.143349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:35071)
0929 03:01:58.144025 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:58.144125 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":502,"threads_started":1}
I20260929 03:01:58.158882 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:58.158150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:48425)
0929 03:01:58.158616 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:58.158714 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
I20260929 03:01:58.159906 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:58.159248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:58483)
0929 03:01:58.159676 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:58.159751 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":248,"threads_started":1}
I20260929 03:01:59.146067 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.145226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:33901)
0929 03:01:59.145782 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.145901 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":270,"threads_started":1}
I20260929 03:01:59.160319 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.159661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:57831)
0929 03:01:59.160067 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.160157 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":194,"threads_started":1}
I20260929 03:01:59.161264 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.160616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:55195)
0929 03:01:59.161021 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.161107 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":215,"threads_started":1}
I20260929 03:01:59.806756 10377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:59.806974 10377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:59.807699 10377 leader_election.cc:288] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387)
I20260929 03:01:59.808643 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.807871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54310)
0929 03:01:59.808333 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.808461 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260929 03:01:59.808889 9643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7947e4e145c84b2499b48de3cf2671ed" is_pre_election: true
I20260929 03:01:59.809105 9643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e8bdcade5c42afa00caabed9279a14 in term 2.
I20260929 03:01:59.809518 9809 leader_election.cc:302] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7947e4e145c84b2499b48de3cf2671ed, e0e8bdcade5c42afa00caabed9279a14; no voters:
I20260929 03:01:59.809767 10377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 03:01:59.809870 10377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:59.810024 10377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:01:59.810235 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.809868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54322)
0929 03:01:59.810019 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.810088 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:01:59.810398 9809 leader_election.cc:334] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:59.812425 10377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:59.812984 10377 leader_election.cc:288] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 election: Requested vote from peers a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357), 7947e4e145c84b2499b48de3cf2671ed (127.2.205.189:39387)
I20260929 03:01:59.813496 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.813118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54334)
0929 03:01:59.813249 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.813354 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:01:59.813699 9643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e8bdcade5c42afa00caabed9279a14" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7947e4e145c84b2499b48de3cf2671ed"
I20260929 03:01:59.813882 9643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:01:59.814323 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.814026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54342)
0929 03:01:59.814157 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.814227 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:01:59.814482 9809 leader_election.cc:334] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
I20260929 03:01:59.815964 9643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e8bdcade5c42afa00caabed9279a14 in term 3.
I20260929 03:01:59.816367 9809 leader_election.cc:302] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7947e4e145c84b2499b48de3cf2671ed, e0e8bdcade5c42afa00caabed9279a14; no voters: a31052ca45b74cbc827b973e04cec020
I20260929 03:01:59.816608 10377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:01:59.817106 10377 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [term 3 LEADER]: Becoming Leader. State: Replica: e0e8bdcade5c42afa00caabed9279a14, State: Running, Role: LEADER
I20260929 03:01:59.817481 10377 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } }
I20260929 03:01:59.818948 10380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e8bdcade5c42afa00caabed9279a14. Latest consensus state: current_term: 3 leader_uuid: "e0e8bdcade5c42afa00caabed9279a14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:59.819228 10380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:59.819818 10382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:01:59.823428 10382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:01:59.824225 10382 catalog_manager.cc:1259] Loaded cluster ID: dc9d60ceb7a04bcc8109910b44efa848
I20260929 03:01:59.824301 10382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:01:59.825495 10382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:01:59.826260 10382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Loaded TSK: 0
I20260929 03:01:59.827119 10382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:01:59.881042 9821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:01:59.888069 9821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:01:59.892338 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.891962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54354)
0929 03:01:59.892094 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.892181 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:01:59.892724 9643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [term 3 FOLLOWER]: Refusing update from remote peer e0e8bdcade5c42afa00caabed9279a14: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 03:01:59.893373 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.893009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54362)
0929 03:01:59.893162 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.893255 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:01:59.893471 10377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 03:01:59.893785 9809 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 -> Peer a31052ca45b74cbc827b973e04cec020 (127.2.205.190:40357): Couldn't send request to peer a31052ca45b74cbc827b973e04cec020. Status: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:01:59.895985 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e8bdcade5c42afa00caabed9279a14. Latest consensus state: current_term: 3 leader_uuid: "e0e8bdcade5c42afa00caabed9279a14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:59.896210 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:59.898353 10380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e0e8bdcade5c42afa00caabed9279a14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:59.898633 10380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14 [sys.catalog]: This master's current role is: LEADER
I20260929 03:01:59.899013 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e0e8bdcade5c42afa00caabed9279a14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a31052ca45b74cbc827b973e04cec020" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40357 } } peers { permanent_uuid: "7947e4e145c84b2499b48de3cf2671ed" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 39387 } attrs { promote: false } } peers { permanent_uuid: "e0e8bdcade5c42afa00caabed9279a14" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 35011 } attrs { promote: false } } }
I20260929 03:01:59.899262 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947e4e145c84b2499b48de3cf2671ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:01:59.916106 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.906890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:39237 (local address 127.0.0.1:46512)
0929 03:01:59.907051 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.907072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:59.907205 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:59.908329 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:59.908334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:59.908360 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:59.908706 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.908719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.911291 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.911298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.913492 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.913510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.915172 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.915178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.915382 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.915429 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:01:59.915521 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:01:59.915733 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:01:59.916126 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.907021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.2.205.131:39237)
0929 03:01:59.908021 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:59.908028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:59.908047 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:01:59.908120 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:59.908126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:59.908202 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:59.908308 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:59.909913 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.911116 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.913695 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.914986 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.915122 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.915247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:59.915556 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 03:01:59.915704 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":784,"threads_started":1}
I20260929 03:01:59.917619 10225 tablet_service.cc:1467] Processing CreateTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 (DEFAULT_TABLE table=test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:59.918150 10225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf0bc4d6d64492f96f205c0c4416c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:59.925432 10395 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Bootstrap starting.
I20260929 03:01:59.927068 10395 tablet_bootstrap.cc:654] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:59.928082 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.908198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:36897 (local address 127.0.0.1:52038)
0929 03:01:59.909255 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.909272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:59.909393 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:59.921402 (+ 12009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:59.921407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:59.921438 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:59.922010 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.922023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.923301 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.923306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.925555 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.925572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.927175 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.927181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.927372 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.927428 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:01:59.927540 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:01:59.927679 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":158,"thread_start_us":792,"threads_started":1}
I20260929 03:01:59.928107 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.907731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.2.205.129:37891)
0929 03:01:59.920896 (+ 13165us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:59.920903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:59.920926 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:01:59.921004 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:59.921010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:59.921087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:59.921204 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:59.921954 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.922992 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.925749 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.926964 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.927080 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.927182 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:59.927690 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 03:01:59.927827 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13034,"thread_start_us":12899,"threads_started":1}
I20260929 03:01:59.928866 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.907343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37891 (local address 127.0.0.1:57036)
0929 03:01:59.907930 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.907950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:01:59.908037 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:01:59.921266 (+ 13229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:01:59.921271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:01:59.921306 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:01:59.921726 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.921739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.923164 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.923170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.925562 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:01:59.925572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.927366 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.927370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:01:59.927513 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.927543 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:01:59.928346 (+ 803us) client_negotiation.cc:240] Negotiation successful
0929 03:01:59.928476 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":95,"thread_start_us":539,"threads_started":1}
I20260929 03:01:59.929361 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.908339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.2.205.130:36897)
0929 03:01:59.921089 (+ 12750us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:59.921094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:59.921112 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:01:59.921176 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:59.921181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:59.921252 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:59.921341 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:01:59.922192 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.923175 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.925746 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.926969 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.927101 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.927188 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:59.927586 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 03:01:59.927738 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12640,"thread_start_us":12551,"threads_started":1}
I20260929 03:01:59.929801 9935 tablet_service.cc:1467] Processing CreateTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 (DEFAULT_TABLE table=test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:59.930083 10395 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: No bootstrap required, opened a new log
I20260929 03:01:59.930047 10084 tablet_service.cc:1467] Processing CreateTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 (DEFAULT_TABLE table=test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:01:59.930231 10395 ts_tablet_manager.cc:1397] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260929 03:01:59.930269 9935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf0bc4d6d64492f96f205c0c4416c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:59.930459 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf0bc4d6d64492f96f205c0c4416c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:01:59.930843 10395 raft_consensus.cc:348] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.931044 10395 raft_consensus.cc:374] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:59.931102 10395 raft_consensus.cc:729] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798ff083b26940d8b4f387dc23b14b1b, State: Initialized, Role: FOLLOWER
I20260929 03:01:59.931305 10395 consensus_queue.cc:260] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.932520 10395 ts_tablet_manager.cc:1428] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:01:59.938109 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.937180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:47465)
0929 03:01:59.937746 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.937888 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
I20260929 03:01:59.941183 10398 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Bootstrap starting.
I20260929 03:01:59.941214 10399 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Bootstrap starting.
I20260929 03:01:59.942710 10398 tablet_bootstrap.cc:654] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:59.942711 10399 tablet_bootstrap.cc:654] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Neither blocks nor log segments found. Creating new log.
I20260929 03:01:59.944976 10398 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: No bootstrap required, opened a new log
I20260929 03:01:59.945000 10399 tablet_bootstrap.cc:492] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: No bootstrap required, opened a new log
I20260929 03:01:59.945128 10398 ts_tablet_manager.cc:1397] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 03:01:59.945133 10399 ts_tablet_manager.cc:1397] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 03:01:59.945905 10399 raft_consensus.cc:348] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.945905 10398 raft_consensus.cc:348] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.946163 10398 raft_consensus.cc:374] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:59.946163 10399 raft_consensus.cc:374] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:01:59.946260 10399 raft_consensus.cc:729] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954e63b28dce47778bc5ac4b7e19bc00, State: Initialized, Role: FOLLOWER
I20260929 03:01:59.946260 10398 raft_consensus.cc:729] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025271554e054a0a9118bfbaaa9c3005, State: Initialized, Role: FOLLOWER
I20260929 03:01:59.946563 10398 consensus_queue.cc:260] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.946777 10399 consensus_queue.cc:260] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.947067 10403 raft_consensus.cc:482] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:01:59.947254 10403 raft_consensus.cc:504] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.947721 10399 ts_tablet_manager.cc:1428] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:01:59.948052 10403 leader_election.cc:288] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237), 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:59.949030 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf0bc4d6d64492f96f205c0c4416c18" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798ff083b26940d8b4f387dc23b14b1b" is_pre_election: true
I20260929 03:01:59.949190 10104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf0bc4d6d64492f96f205c0c4416c18" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025271554e054a0a9118bfbaaa9c3005" is_pre_election: true
I20260929 03:01:59.949286 10245 raft_consensus.cc:2454] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 0.
I20260929 03:01:59.949393 10104 raft_consensus.cc:2454] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 0.
I20260929 03:01:59.949843 9890 leader_election.cc:302] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025271554e054a0a9118bfbaaa9c3005, 954e63b28dce47778bc5ac4b7e19bc00; no voters:
I20260929 03:01:59.950090 10403 raft_consensus.cc:2781] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:01:59.950192 10403 raft_consensus.cc:482] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:01:59.950271 10403 raft_consensus.cc:3037] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:59.950277 10398 ts_tablet_manager.cc:1428] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260929 03:01:59.950700 10000 heartbeater.cc:503] Master 127.2.205.188:35011 was elected leader, sending a full tablet report...
I20260929 03:01:59.951748 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.950788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:54699)
0929 03:01:59.951310 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.951480 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
I20260929 03:01:59.953143 10403 raft_consensus.cc:504] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.953599 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.948684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:38045)
0929 03:01:59.953153 (+ 4469us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.953320 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4344,"thread_start_us":4212,"threads_started":1}
I20260929 03:01:59.954653 10403 leader_election.cc:288] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 election: Requested vote from peers 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237), 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897)
I20260929 03:01:59.954757 10104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf0bc4d6d64492f96f205c0c4416c18" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025271554e054a0a9118bfbaaa9c3005"
I20260929 03:01:59.954705 10245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf0bc4d6d64492f96f205c0c4416c18" candidate_uuid: "954e63b28dce47778bc5ac4b7e19bc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798ff083b26940d8b4f387dc23b14b1b"
I20260929 03:01:59.954934 10245 raft_consensus.cc:3037] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:59.954952 10104 raft_consensus.cc:3037] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:01:59.956895 10149 heartbeater.cc:503] Master 127.2.205.188:35011 was elected leader, sending a full tablet report...
I20260929 03:01:59.957748 10104 raft_consensus.cc:2454] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 1.
I20260929 03:01:59.957749 10245 raft_consensus.cc:2454] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954e63b28dce47778bc5ac4b7e19bc00 in term 1.
I20260929 03:01:59.958175 9889 leader_election.cc:302] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 798ff083b26940d8b4f387dc23b14b1b, 954e63b28dce47778bc5ac4b7e19bc00; no voters:
I20260929 03:01:59.958873 10403 raft_consensus.cc:2781] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:01:59.958994 10403 raft_consensus.cc:686] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 LEADER]: Becoming Leader. State: Replica: 954e63b28dce47778bc5ac4b7e19bc00, State: Running, Role: LEADER
I20260929 03:01:59.959226 10403 consensus_queue.cc:237] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } }
I20260929 03:01:59.959786 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.958372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:48883)
0929 03:01:59.959555 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.959640 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1084}
I20260929 03:01:59.960968 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.960499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:33449)
0929 03:01:59.960684 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.960828 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 03:01:59.961706 9821 catalog_manager.cc:5495] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "954e63b28dce47778bc5ac4b7e19bc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "954e63b28dce47778bc5ac4b7e19bc00" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 } health_report { overall_health: UNKNOWN } } }
I20260929 03:01:59.979079 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.974514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.2.205.129:37891)
0929 03:01:59.974683 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:01:59.974689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:01:59.974709 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:01:59.974898 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:01:59.974902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:01:59.974963 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:01:59.975048 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:01:59.975405 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.976100 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.977510 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:01:59.977920 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:01:59.978038 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:01:59.978427 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:01:59.978535 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:01:59.978608 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:01:59.978743 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:01:59.978819 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:01:59.982155 10245 raft_consensus.cc:1261] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Refusing update from remote peer 954e63b28dce47778bc5ac4b7e19bc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:59.982126 10104 raft_consensus.cc:1261] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Refusing update from remote peer 954e63b28dce47778bc5ac4b7e19bc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:01:59.982788 10403 consensus_queue.cc:1035] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025271554e054a0a9118bfbaaa9c3005" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:59.983060 10405 consensus_queue.cc:1035] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ff083b26940d8b4f387dc23b14b1b" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:01:59.984537 10290 heartbeater.cc:503] Master 127.2.205.188:35011 was elected leader, sending a full tablet report...
I20260929 03:01:59.985271 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.984716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:49143)
0929 03:01:59.984984 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.985091 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 03:01:59.985622 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.985144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:43967)
0929 03:01:59.985312 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.985435 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 03:01:59.992163 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.991655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.131:43585)
0929 03:01:59.991867 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.991970 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 03:01:59.992359 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.991794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.129:51857)
0929 03:01:59.992084 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.992183 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 03:01:59.995139 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:01:59.994555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.2.205.130:44513)
0929 03:01:59.994872 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 03:01:59.994966 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260929 03:01:59.995707 10411 mvcc.cc:205] Tried to move back new op lower bound from 7334506168240705536 to 7334506168155078656. Current Snapshot: MvccSnapshot[applied={T|T < 7334506168240705536}]
I20260929 03:02:00.000583 9821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:00.000881 9821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:00.006449 9821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0e8bdcade5c42afa00caabed9279a14: Sending DeleteTablet for 3 replicas of tablet aaf0bc4d6d64492f96f205c0c4416c18
I20260929 03:02:00.007930 10084 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:00 UTC) from {username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:52038
I20260929 03:02:00.007901 9935 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:00 UTC) from {username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:57036
I20260929 03:02:00.008606 10225 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf0bc4d6d64492f96f205c0c4416c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:00 UTC) from {username='oryx', principal='oryx/127.2.205.188@KRBTEST.COM'} at 127.0.0.1:46512
I20260929 03:02:00.009109 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9799
I20260929 03:02:00.012876 10420 tablet_replica.cc:330] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: stopping tablet replica
I20260929 03:02:00.013165 10420 raft_consensus.cc:2229] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:00.013612 10420 raft_consensus.cc:2258] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:00.014780 10420 ts_tablet_manager.cc:1905] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:00.018980 10419 tablet_replica.cc:330] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: stopping tablet replica
I20260929 03:02:00.019349 10419 raft_consensus.cc:2229] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:00.019748 10419 raft_consensus.cc:2258] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:00.021129 10418 tablet_replica.cc:330] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: stopping tablet replica
I20260929 03:02:00.021369 10418 raft_consensus.cc:2229] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:00.021497 10419 ts_tablet_manager.cc:1905] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:00.021838 10418 raft_consensus.cc:2258] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:00.022221 10420 ts_tablet_manager.cc:1918] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:00.022426 10420 log.cc:1196] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/aaf0bc4d6d64492f96f205c0c4416c18
I20260929 03:02:00.022943 10420 ts_tablet_manager.cc:1939] T aaf0bc4d6d64492f96f205c0c4416c18 P 798ff083b26940d8b4f387dc23b14b1b: Deleting consensus metadata
I20260929 03:02:00.023077 10418 ts_tablet_manager.cc:1905] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:00.024115 9807 catalog_manager.cc:4841] TS 798ff083b26940d8b4f387dc23b14b1b (127.2.205.131:39237): tablet aaf0bc4d6d64492f96f205c0c4416c18 (table test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]) successfully deleted
I20260929 03:02:00.027456 10419 ts_tablet_manager.cc:1918] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:00.027570 10419 log.cc:1196] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/aaf0bc4d6d64492f96f205c0c4416c18
I20260929 03:02:00.028082 10419 ts_tablet_manager.cc:1939] T aaf0bc4d6d64492f96f205c0c4416c18 P 025271554e054a0a9118bfbaaa9c3005: Deleting consensus metadata
I20260929 03:02:00.028981 9808 catalog_manager.cc:4841] TS 025271554e054a0a9118bfbaaa9c3005 (127.2.205.130:36897): tablet aaf0bc4d6d64492f96f205c0c4416c18 (table test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]) successfully deleted
I20260929 03:02:00.029345 10418 ts_tablet_manager.cc:1918] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:00.029477 10418 log.cc:1196] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/aaf0bc4d6d64492f96f205c0c4416c18
I20260929 03:02:00.029873 10418 ts_tablet_manager.cc:1939] T aaf0bc4d6d64492f96f205c0c4416c18 P 954e63b28dce47778bc5ac4b7e19bc00: Deleting consensus metadata
I20260929 03:02:00.030840 9808 catalog_manager.cc:4841] TS 954e63b28dce47778bc5ac4b7e19bc00 (127.2.205.129:37891): tablet aaf0bc4d6d64492f96f205c0c4416c18 (table test-table [id=63c7b6c0236d4ff5ba239ebea7f2d465]) successfully deleted
I20260929 03:02:00.035259 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:00.010190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.2.205.129:37891)
0929 03:02:00.010401 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:00.010408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:00.020432 (+ 10024us) server_negotiation.cc:407] Connection header received
0929 03:02:00.020788 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:00.020794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:00.020903 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:00.021015 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:00.023367 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.024451 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.026273 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.026929 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.027063 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:00.030321 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:00.030338 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:00.030354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:00.030390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:00.033040 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.033670 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.033677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.033679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.033806 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.034092 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.034095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.034097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.034413 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:00.034624 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:00.034861 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 03:02:00.035014 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:02:00.035862 9995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:00.154546 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9799
I20260929 03:02:00.175041 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10012
I20260929 03:02:00.188618 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:00.175916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.2.205.130:36897)
0929 03:02:00.176087 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:00.176091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:00.176109 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:02:00.176210 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:00.176214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:00.176277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:00.176361 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:00.177890 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.178592 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.179888 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.180359 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.180469 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:00.182948 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:00.182964 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:00.182981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:00.183012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:00.185985 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.186663 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.186670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.186673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.186812 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.187169 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.187173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.187175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.187534 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:00.187826 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:00.188093 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 03:02:00.188322 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:02:00.189325 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:00.282696 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10012
I20260929 03:02:00.296066 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10162
I20260929 03:02:00.309082 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:00.296823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.2.205.131:39237)
0929 03:02:00.296992 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:00.296997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:00.297056 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:02:00.297275 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:00.297279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:00.297408 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:00.297602 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:00.298948 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.299638 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.300927 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:00.301356 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:00.301463 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:00.303908 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:00.303932 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:00.303948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:00.303986 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:00.306609 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.307242 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.307250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.307253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.307386 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:00.307665 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:00.307668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:00.307670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:00.308056 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:00.308275 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:00.308531 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 03:02:00.308707 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:02:00.309759 10285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:00.366962 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:00.366541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54376)
0929 03:02:00.366711 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:00.366820 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:02:00.368048 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:00.367672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40357 (local address 127.0.0.1:54378)
0929 03:02:00.367844 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:00.367920 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:02:00.408079 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10162
I20260929 03:02:00.422405 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9498
I20260929 03:02:00.423520 9664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:00.538888 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9498
I20260929 03:02:00.555632 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 9598
I20260929 03:02:00.556769 9852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:00.676440 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 9598
2026-09-29T03:02:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11300 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:02:00 dist-test-slave-530n krb5kdc[10429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:02:00 dist-test-slave-530n krb5kdc[10429](info): set up 2 sockets
Sep 29 03:02:00 dist-test-slave-530n krb5kdc[10429](info): commencing operation
krb5kdc: starting...
W20260929 03:02:01.681603 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:02:01 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab.
Entry for principal oryx/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab.
I20260929 03:02:01.891728 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38399
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab
--principal=oryx/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:02.215054 10442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:02.215407 10442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:02.215698 10442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:02.224260 10442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:02.224431 10442 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:02:02.224502 10442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:02.224606 10442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:02.224673 10442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:02.224786 10442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:02.235255 10442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:38399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.190.keytab
--principal=oryx/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:02.236732 10442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:02.238374 10442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:02.251129 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:02.251185 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:02.251164 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:02.251853 10442 server_base.cc:1034] running on GCE node
I20260929 03:02:02.252619 10442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:02.254019 10442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:02.255235 10442 hybrid_clock.cc:648] HybridClock initialized: now 1790650922255171 us; error 41 us; skew 500 ppm
I20260929 03:02:02.255555 10442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:02 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650922, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:02.261723 10442 init.cc:377] Logged in from keytab as oryx/127.2.205.190@KRBTEST.COM (short username oryx)
I20260929 03:02:02.263090 10442 webserver.cc:460] Webserver started at http://127.2.205.190:32777/ using document root <none> and password file <none>
I20260929 03:02:02.263736 10442 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:02.263818 10442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:02.264081 10442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:02.266868 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "3ecff5ecf1964f3ba05212d4da427bab"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "98e07fa0b112534e1f0c3cdf7dfdaafb"
server_key_iv: "26566958e951283ddfd1ded752b2a8b8"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.267565 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "3ecff5ecf1964f3ba05212d4da427bab"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "98e07fa0b112534e1f0c3cdf7dfdaafb"
server_key_iv: "26566958e951283ddfd1ded752b2a8b8"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.272151 10442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 03:02:02.275241 10457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:02.276499 10442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:02.276631 10442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "3ecff5ecf1964f3ba05212d4da427bab"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "98e07fa0b112534e1f0c3cdf7dfdaafb"
server_key_iv: "26566958e951283ddfd1ded752b2a8b8"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.276847 10442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:02.303823 10442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:02.307848 10442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:02.308084 10442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:02.328215 10442 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:38399
I20260929 03:02:02.328234 10518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:38399 every 8 connection(s)
I20260929 03:02:02.329630 10442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:02.332806 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10442
I20260929 03:02:02.333030 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:02:02.333844 2870 external_mini_cluster.cc:1428] Setting key b2ca558a9b387964352616f557d780d1
I20260929 03:02:02.334137 10519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:02.348858 10519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Bootstrap starting.
Sep 29 03:02:02 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:02.352224 10519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:02.353271 10519 log.cc:824] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:02.355845 10519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: No bootstrap required, opened a new log
I20260929 03:02:02.356987 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:02.335899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.2.205.190:38399)
0929 03:02:02.336614 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:02.336626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:02.336670 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:02:02.338899 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:02.338923 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:02.339376 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:02.339797 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:02.341706 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.343104 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.344868 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.345624 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.345762 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:02.350267 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:02.350276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:02.350291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:02.350335 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:02.353246 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.353879 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.353886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.353892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.354056 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.354417 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.354421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.354423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.354908 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:02.355173 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:02.355593 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 03:02:02.356007 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260929 03:02:02.363288 10519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } }
I20260929 03:02:02.363597 10519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:02.363695 10519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecff5ecf1964f3ba05212d4da427bab, State: Initialized, Role: FOLLOWER
I20260929 03:02:02.364226 10519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } }
I20260929 03:02:02.364444 10519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:02.364562 10519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:02.364699 10519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:02.366724 10519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } }
I20260929 03:02:02.367201 10519 leader_election.cc:302] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ecff5ecf1964f3ba05212d4da427bab; no voters:
I20260929 03:02:02.367707 10519 leader_election.cc:288] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:02:02.367820 10524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:02.368793 10524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [term 1 LEADER]: Becoming Leader. State: Replica: 3ecff5ecf1964f3ba05212d4da427bab, State: Running, Role: LEADER
I20260929 03:02:02.369309 10524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } }
I20260929 03:02:02.369647 10519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:02.371807 10525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ecff5ecf1964f3ba05212d4da427bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } } }
I20260929 03:02:02.372223 10525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:02.371825 10526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ecff5ecf1964f3ba05212d4da427bab. Latest consensus state: current_term: 1 leader_uuid: "3ecff5ecf1964f3ba05212d4da427bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecff5ecf1964f3ba05212d4da427bab" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38399 } } }
I20260929 03:02:02.372555 10526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:02.374334 10532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:02.379534 10532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:02.389508 10532 catalog_manager.cc:1347] Generated new cluster ID: c7647bf0010748a7bef921209aca7b07
I20260929 03:02:02.389631 10532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:02.408226 10532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:02:02.410282 10532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:02.430423 10532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Generated new TSK 0
I20260929 03:02:02.431607 10532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
I20260929 03:02:02.480455 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38399
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:02.804247 10545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:02.804559 10545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:02.804662 10545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:02.812105 10545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:02.812227 10545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:02.812321 10545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:02:02.812398 10545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:02.822563 10545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:02.823843 10545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:02.825204 10545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:02.838429 10550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:02.838466 10553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:02.838512 10551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:02.842374 10545 server_base.cc:1034] running on GCE node
I20260929 03:02:02.843040 10545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:02.843981 10545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:02.845206 10545 hybrid_clock.cc:648] HybridClock initialized: now 1790650922845137 us; error 42 us; skew 500 ppm
I20260929 03:02:02.845522 10545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:02 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650922, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:02.848573 10545 init.cc:377] Logged in from keytab as oryx/127.2.205.129@KRBTEST.COM (short username oryx)
I20260929 03:02:02.850085 10545 webserver.cc:460] Webserver started at http://127.2.205.129:44793/ using document root <none> and password file <none>
I20260929 03:02:02.850776 10545 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:02.850871 10545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:02.851172 10545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:02.854667 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "ea257bc77c5c9ab5a99bbe6ac5343043"
server_key_iv: "effe04711674756dd36c29e4c94c2494"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.855417 10545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "ea257bc77c5c9ab5a99bbe6ac5343043"
server_key_iv: "effe04711674756dd36c29e4c94c2494"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.860280 10545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:02:02.863706 10560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:02.865043 10545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:02.865195 10545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
format_stamp: "Formatted at 2026-09-29 03:02:02 on dist-test-slave-530n"
server_key: "ea257bc77c5c9ab5a99bbe6ac5343043"
server_key_iv: "effe04711674756dd36c29e4c94c2494"
server_key_version: "encryptionkey@0"
I20260929 03:02:02.865384 10545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:02.879735 10545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:02.883873 10545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:02.884120 10545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:02.886811 10545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:02.886894 10545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:02.887002 10545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:02.887058 10545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:02:02 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650922, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:02.913926 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:02.888096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.129:39183)
0929 03:02:02.889007 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:02.889065 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:02.891216 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:02.892206 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:02.892215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:02.893178 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:02.894167 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:02.894201 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.895608 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.895614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:02.898421 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:02.898446 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.899515 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.899525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:02.899913 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:02.900833 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:02.900863 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:02.904267 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:02.908057 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:02.908069 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:02.908100 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:02.908899 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:02.909345 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:02.909350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:02.909354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:02.909620 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:02.910421 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:02.910430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:02.910984 (+ 554us) client_negotiation.cc:769] Sending connection context
0929 03:02:02.911416 (+ 432us) client_negotiation.cc:240] Negotiation successful
0929 03:02:02.912085 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":367,"threads_started":1}
I20260929 03:02:02.915182 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:02.887500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:39183 (local address 127.2.205.190:38399)
0929 03:02:02.889798 (+ 2298us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:02.889804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:02.889828 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:02:02.891612 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:02.891619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:02.891723 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:02.891845 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:02.894421 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.895404 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.898646 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.899212 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.899370 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:02.904673 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:02.904683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:02.904689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:02.904784 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:02.907839 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.909066 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.909071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.909073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.909220 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.909788 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.909795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.909798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.910252 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:02.910442 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:02.914612 (+ 4170us) server_negotiation.cc:299] Negotiation successful
0929 03:02:02.914884 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2175,"thread_start_us":2077,"threads_started":1}
I20260929 03:02:02.928484 10545 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37135
I20260929 03:02:02.928462 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37135 every 8 connection(s)
I20260929 03:02:02.930070 10545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:02.931207 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10545
I20260929 03:02:02.931398 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:02:02.932189 2870 external_mini_cluster.cc:1428] Setting key c00f51ed5676b09f83b19440ef1e1a69
I20260929 03:02:02.932390 10681 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38399
I20260929 03:02:02.932649 10681 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:02.934022 10681 heartbeater.cc:511] Master 127.2.205.190:38399 requested a full tablet report, sending...
I20260929 03:02:02.936604 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:02.922251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.129:38613)
0929 03:02:02.922857 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:02.922873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:02.923036 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:02.923430 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:02.923436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:02.924061 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:02.924476 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:02.924491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.925814 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.925819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:02.927237 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:02.927253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.928178 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.928185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:02.928368 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:02.929050 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:02.929068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:02.929851 (+ 783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:02.933611 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:02.933617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:02.933620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:02.934188 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:02.934641 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:02.934646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:02.934649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:02.934804 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:02.935489 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:02.935495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:02.935683 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 03:02:02.935795 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:02:02.936146 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":324,"threads_started":1}
I20260929 03:02:02.936697 10474 ts_manager.cc:194] Registered new tserver with Master: b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135)
I20260929 03:02:02.937965 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:02.922194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:38613 (local address 127.2.205.190:38399)
0929 03:02:02.922380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:02.922384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:02.922984 (+ 600us) server_negotiation.cc:407] Connection header received
0929 03:02:02.923164 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:02.923170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:02.923260 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:02.923366 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:02.924646 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.925620 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.927411 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:02.927976 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:02.928098 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:02.930064 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:02.930069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:02.930073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:02.930111 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:02.933433 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.934343 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.934348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.934351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.934512 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:02.934961 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:02.934966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:02.934969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:02.935344 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:02.937312 (+ 1968us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:02.937446 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:02:02.937662 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:02:02.939100 10474 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:39183
WARNING: no policy specified for oryx/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
I20260929 03:02:02.980206 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38399
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:03.307452 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:03.307771 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:03.307911 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:03.315311 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:03.315466 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:03.315591 10685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:02:03.315702 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:03.326102 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:03.327484 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:03.329006 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:03.344321 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:03.344522 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:03.344938 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:03.345224 10685 server_base.cc:1034] running on GCE node
I20260929 03:02:03.345857 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:03.346786 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:03.348044 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790650923347981 us; error 39 us; skew 500 ppm
I20260929 03:02:03.348335 10685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:03.351413 10685 init.cc:377] Logged in from keytab as oryx/127.2.205.130@KRBTEST.COM (short username oryx)
I20260929 03:02:03.352731 10685 webserver.cc:460] Webserver started at http://127.2.205.130:38211/ using document root <none> and password file <none>
I20260929 03:02:03.353452 10685 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:03.353551 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:03.353852 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:03.356659 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "91de360d29344e4fa720f0279162ba3e"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d3d52d41e2159f1518573f3cd20316c7"
server_key_iv: "b9298ef9a105b4c4e02abd9fc9b5d88c"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.357429 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "91de360d29344e4fa720f0279162ba3e"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d3d52d41e2159f1518573f3cd20316c7"
server_key_iv: "b9298ef9a105b4c4e02abd9fc9b5d88c"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.362268 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:02:03.365612 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:03.366868 10685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:02:03.367022 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "91de360d29344e4fa720f0279162ba3e"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d3d52d41e2159f1518573f3cd20316c7"
server_key_iv: "b9298ef9a105b4c4e02abd9fc9b5d88c"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.367215 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:03.378298 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:03.382884 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:03.383153 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:03.386003 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:03.386096 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:03.386216 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:03.386273 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:03.409294 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.386563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:56027 (local address 127.2.205.190:38399)
0929 03:02:03.386746 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:03.386753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:03.388311 (+ 1558us) server_negotiation.cc:407] Connection header received
0929 03:02:03.390836 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:03.390844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:03.390943 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:03.391029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:03.393715 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.394707 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.397112 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.397693 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.397814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.402519 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:03.402526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:03.402531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:03.402577 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:03.405369 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.406322 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.406329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.406333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.406509 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.407060 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.407065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.407068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.407414 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:03.407581 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:03.408688 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0929 03:02:03.408952 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:02:03.411121 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.387234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.130:56027)
0929 03:02:03.388107 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:03.388163 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:03.390434 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:03.391621 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:03.391637 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:03.392498 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:03.393498 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.393532 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.394938 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.394947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.396935 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.396951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.397956 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.397964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.398295 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.399167 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:03.399207 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:03.402251 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:03.405502 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.405512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.405533 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.406157 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.406640 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.406645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.406650 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.406936 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.407570 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:03.407579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:03.408173 (+ 594us) client_negotiation.cc:769] Sending connection context
0929 03:02:03.408566 (+ 393us) client_negotiation.cc:240] Negotiation successful
0929 03:02:03.409257 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":356,"threads_started":1}
I20260929 03:02:03.428485 10685 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:41895
I20260929 03:02:03.428515 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:41895 every 8 connection(s)
I20260929 03:02:03.429886 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:03.430688 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10685
I20260929 03:02:03.430856 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:02:03.431439 2870 external_mini_cluster.cc:1428] Setting key f9ff076bc83fb53f327d1516f8293ced
I20260929 03:02:03.433054 10820 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38399
I20260929 03:02:03.433178 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.417321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.130:54309)
0929 03:02:03.418090 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:03.418110 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:03.418246 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:03.418754 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:03.418759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:03.419439 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:03.419967 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.419985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.421391 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.421397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.422959 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.422978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.424238 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.424248 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.424448 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.425216 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:03.425244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:03.426602 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:03.429626 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.429632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.429635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.430136 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.430971 (+ 835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.430976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.430980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.431144 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.431891 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:03.431897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:03.432119 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 03:02:03.432281 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 03:02:03.432651 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":440,"threads_started":1}
I20260929 03:02:03.434535 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.417315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:54309 (local address 127.2.205.190:38399)
0929 03:02:03.417529 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:03.417535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:03.418191 (+ 656us) server_negotiation.cc:407] Connection header received
0929 03:02:03.418370 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:03.418375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:03.418453 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:03.418564 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:03.420167 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.421253 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.423134 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.423826 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.423955 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.426788 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:03.426793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:03.426796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:03.426830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:03.429500 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.430252 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.430256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.430258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.430824 (+ 566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.431295 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.431299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.431302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.431762 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:03.431914 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:03.433898 (+ 1984us) server_negotiation.cc:299] Negotiation successful
0929 03:02:03.434169 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:02:03.439496 10820 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:03.440358 10820 heartbeater.cc:511] Master 127.2.205.190:38399 requested a full tablet report, sending...
I20260929 03:02:03.441880 10473 ts_manager.cc:194] Registered new tserver with Master: 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:03.442709 10473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:56027
WARNING: no policy specified for oryx/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
I20260929 03:02:03.477741 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38399
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:03.798727 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:03.799032 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:03.799131 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:03.806525 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:03.806650 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:03.806738 10824 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:02:03.806816 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:03.816987 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.205.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:03.818430 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:03.819834 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:03.832763 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:03.832829 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:03.832808 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:03.833688 10824 server_base.cc:1034] running on GCE node
I20260929 03:02:03.834388 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:03.835354 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:03.836575 10824 hybrid_clock.cc:648] HybridClock initialized: now 1790650923836509 us; error 40 us; skew 500 ppm
I20260929 03:02:03.836915 10824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:03.840019 10824 init.cc:377] Logged in from keytab as oryx/127.2.205.131@KRBTEST.COM (short username oryx)
I20260929 03:02:03.841614 10824 webserver.cc:460] Webserver started at http://127.2.205.131:35379/ using document root <none> and password file <none>
I20260929 03:02:03.842339 10824 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:03.842442 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:03.842754 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:03.845702 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d19de37236c99e4ccfa2d4fe5648ad21"
server_key_iv: "6fa6c31e6b9df5db17a2dd6cffd1eb90"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.846509 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d19de37236c99e4ccfa2d4fe5648ad21"
server_key_iv: "6fa6c31e6b9df5db17a2dd6cffd1eb90"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.851454 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:02:03.854889 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:03.856472 10824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:03.856632 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
format_stamp: "Formatted at 2026-09-29 03:02:03 on dist-test-slave-530n"
server_key: "d19de37236c99e4ccfa2d4fe5648ad21"
server_key_iv: "6fa6c31e6b9df5db17a2dd6cffd1eb90"
server_key_version: "encryptionkey@0"
I20260929 03:02:03.856884 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:03.881313 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:03.885860 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:03.886142 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:03.888965 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:03.889060 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:03.889182 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:03.889238 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:03.914539 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.889650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:46831 (local address 127.2.205.190:38399)
0929 03:02:03.889862 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:03.889868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:03.891504 (+ 1636us) server_negotiation.cc:407] Connection header received
0929 03:02:03.894031 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:03.894038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:03.894154 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:03.894289 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:03.896729 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.897928 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.900399 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.901097 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.901255 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.906901 (+ 5646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:03.906909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:03.906914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:03.906965 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:03.910294 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.911480 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.911486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.911490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.911658 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.912364 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.912370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.912373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.912836 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:03.912982 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:03.913863 (+ 881us) server_negotiation.cc:299] Negotiation successful
0929 03:02:03.914161 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:02:03.915663 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.890340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.131:46831)
0929 03:02:03.891274 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:03.891335 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:03.893615 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:03.894630 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:03.894644 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:03.895402 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:03.896432 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.896457 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.898138 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.898142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.900209 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.900224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.901419 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.901428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.901815 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.902782 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:03.902823 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:03.906579 (+ 3756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:03.910466 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.910476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.910502 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.911250 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.911821 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.911828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.911834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.912193 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.912995 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:03.913004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:03.913469 (+ 465us) client_negotiation.cc:769] Sending connection context
0929 03:02:03.913751 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 03:02:03.914364 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":380,"threads_started":1}
I20260929 03:02:03.934667 10824 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:43533
I20260929 03:02:03.934684 10958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:43533 every 8 connection(s)
I20260929 03:02:03.936163 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:03.938138 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 10824
I20260929 03:02:03.938318 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:02:03.938714 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.922177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.2.205.131:57773)
0929 03:02:03.922881 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:03.922906 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:03.923113 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:03.923557 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:03.923562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:03.924236 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:03.924684 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.924701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.925755 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.925764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.927886 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:03.927909 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.929192 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.929201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:03.929405 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.930175 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:03.930202 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:03.931716 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:03.935555 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.935561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.935565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.936094 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.936508 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:03.936514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:03.936517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:03.936696 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:03.937493 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:03.937497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:03.937897 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 03:02:03.938042 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 03:02:03.938367 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":390,"threads_started":1}
I20260929 03:02:03.938777 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:03.922122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:57773 (local address 127.2.205.190:38399)
0929 03:02:03.922342 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:03.922348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:03.922997 (+ 649us) server_negotiation.cc:407] Connection header received
0929 03:02:03.923259 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:03.923266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:03.923371 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:03.923478 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:03.924872 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.925620 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.928067 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.928655 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.928823 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:03.932003 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:03.932010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:03.932015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:03.932060 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:03.935391 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.936216 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.936221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.936224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.936377 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:03.936861 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:03.936866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:03.936870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:03.937248 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:03.937424 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:03.938208 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 03:02:03.938470 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:02:03.938879 2870 external_mini_cluster.cc:1428] Setting key fbb7c9581ce3b466e588fed47c62870b
I20260929 03:02:03.939779 10959 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38399
I20260929 03:02:03.940037 10959 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:03.940834 10959 heartbeater.cc:511] Master 127.2.205.190:38399 requested a full tablet report, sending...
I20260929 03:02:03.942278 10474 ts_manager.cc:194] Registered new tserver with Master: ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533)
I20260929 03:02:03.943107 10474 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:46831
I20260929 03:02:03.943629 10681 heartbeater.cc:503] Master 127.2.205.190:38399 was elected leader, sending a full tablet report...
I20260929 03:02:03.953990 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:03 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:04.445474 10820 heartbeater.cc:503] Master 127.2.205.190:38399 was elected leader, sending a full tablet report...
I20260929 03:02:04.945837 10959 heartbeater.cc:503] Master 127.2.205.190:38399 was elected leader, sending a full tablet report...
W20260929 03:02:04.967996 10967 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.972s user 0.003s sys 0.010s
W20260929 03:02:04.969076 10569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:03.990822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.2.205.190:38399)
0929 03:02:03.991052 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:03.991059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:03.991156 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:02:03.991347 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:03.991351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:03.991416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:03.991515 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:03.992824 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.993562 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.994836 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:03.995253 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:03.995381 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:04.968856 (+973475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35920: BlockingRecv error: recv got EOF from 127.0.0.1:35920 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 03:02:04 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:04.985826 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:04.972484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.2.205.190:38399)
0929 03:02:04.972689 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:04.972696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:04.973023 (+ 327us) server_negotiation.cc:407] Connection header received
0929 03:02:04.973156 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:04.973160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:04.973236 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:04.973321 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:04.974587 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:04.975270 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:04.976429 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:04.976862 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:04.976953 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:04.980558 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:04.980563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:04.980567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:04.980602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:04.983245 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:04.983899 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:04.983903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:04.983906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:04.984035 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:04.984386 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:04.984390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:04.984392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:04.984690 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:04.985002 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:04.985272 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 03:02:04.985526 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:02:04.989768 10473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:02:04.996691 10473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:05.036450 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.009337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.2.205.129:37135)
0929 03:02:05.021877 (+ 12540us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.021888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.028201 (+ 6313us) server_negotiation.cc:407] Connection header received
0929 03:02:05.028322 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.028332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.028652 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.028878 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.029887 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.031373 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.034081 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.035326 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.035450 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.035694 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.035975 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.036152 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12357,"thread_start_us":12196,"threads_started":1}
I20260929 03:02:05.036491 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.009158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37135 (local address 127.0.0.1:46044)
0929 03:02:05.027795 (+ 18637us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.027844 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.028104 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.029166 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.029175 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.029207 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.029701 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.029714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.031547 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.031552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.033899 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.033914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.035549 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.035556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.035747 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.035789 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.035902 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.036133 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18344,"thread_start_us":18202,"threads_started":1}
I20260929 03:02:05.037735 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.010406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.2.205.130:41895)
0929 03:02:05.026956 (+ 16550us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.026977 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.029523 (+ 2546us) server_negotiation.cc:407] Connection header received
0929 03:02:05.029841 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.029848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.030131 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.030309 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.031212 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.032473 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.034867 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.036352 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.036504 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.036719 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.037209 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.037391 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16340,"thread_start_us":16572,"threads_started":1}
I20260929 03:02:05.037986 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.009887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41895 (local address 127.0.0.1:45662)
0929 03:02:05.029396 (+ 19509us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.029413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.029638 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.030587 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.030592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.030616 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.030996 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.031008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.032630 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.032636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.034684 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.034702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.036875 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.036882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.037032 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.037066 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.037156 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.037680 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19382,"thread_start_us":18576,"threads_started":1}
I20260929 03:02:05.039793 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.009297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.0.0.1:36886)
0929 03:02:05.028709 (+ 19412us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.028727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.028875 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.032131 (+ 3256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.032136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.032163 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.032547 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.032560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.034553 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.034558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.036523 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.036538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.039021 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.039027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.039203 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.039238 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.039330 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.039480 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19275,"thread_start_us":19792,"threads_started":1}
I20260929 03:02:05.040196 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.009893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.2.205.131:43533)
0929 03:02:05.031521 (+ 21628us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.031530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.031556 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:02:05.031658 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.031665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.031898 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.032110 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.032709 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.034423 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.037347 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.038786 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.038934 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.039172 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.039626 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.039790 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21462,"thread_start_us":21340,"threads_started":1}
I20260929 03:02:05.041368 10611 tablet_service.cc:1467] Processing CreateTablet for tablet dec4942430c840f7a86e4475a6cd48bb (DEFAULT_TABLE table=test-table [id=29db7f9669cb493d889550adf7b3290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.041524 10750 tablet_service.cc:1467] Processing CreateTablet for tablet dec4942430c840f7a86e4475a6cd48bb (DEFAULT_TABLE table=test-table [id=29db7f9669cb493d889550adf7b3290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.042913 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4942430c840f7a86e4475a6cd48bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.042932 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4942430c840f7a86e4475a6cd48bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.043727 10889 tablet_service.cc:1467] Processing CreateTablet for tablet dec4942430c840f7a86e4475a6cd48bb (DEFAULT_TABLE table=test-table [id=29db7f9669cb493d889550adf7b3290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.045049 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4942430c840f7a86e4475a6cd48bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.054566 10984 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Bootstrap starting.
I20260929 03:02:05.057422 10985 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Bootstrap starting.
I20260929 03:02:05.057928 10984 tablet_bootstrap.cc:654] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.058810 10986 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Bootstrap starting.
I20260929 03:02:05.059226 10984 log.cc:824] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:05.061081 10985 tablet_bootstrap.cc:654] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.061859 10986 tablet_bootstrap.cc:654] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.062455 10985 log.cc:824] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:05.062698 10984 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: No bootstrap required, opened a new log
I20260929 03:02:05.062790 10986 log.cc:824] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:05.062958 10984 ts_tablet_manager.cc:1397] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 03:02:05.065227 10985 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: No bootstrap required, opened a new log
I20260929 03:02:05.065483 10985 ts_tablet_manager.cc:1397] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 03:02:05.065598 10986 tablet_bootstrap.cc:492] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: No bootstrap required, opened a new log
I20260929 03:02:05.065842 10986 ts_tablet_manager.cc:1397] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 03:02:05.070672 10984 raft_consensus.cc:348] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.071038 10984 raft_consensus.cc:374] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.071126 10984 raft_consensus.cc:729] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.071727 10984 consensus_queue.cc:260] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.075788 10985 raft_consensus.cc:348] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.075982 10986 raft_consensus.cc:348] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.076253 10985 raft_consensus.cc:374] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.076341 10986 raft_consensus.cc:374] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.076356 10985 raft_consensus.cc:729] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.076452 10986 raft_consensus.cc:729] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.077049 10986 consensus_queue.cc:260] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.077033 10985 consensus_queue.cc:260] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.078969 10986 ts_tablet_manager.cc:1428] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260929 03:02:05.079356 10985 ts_tablet_manager.cc:1428] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260929 03:02:05.081674 10984 ts_tablet_manager.cc:1428] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Time spent starting tablet: real 0.019s user 0.008s sys 0.007s
W20260929 03:02:05.188642 10960 tablet.cc:2369] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:02:05.191340 10682 tablet.cc:2369] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:02:05.193256 10821 tablet.cc:2369] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:02:05.476071 10990 raft_consensus.cc:482] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:05.476402 10990 raft_consensus.cc:504] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.478160 10990 leader_election.cc:288] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:05.485833 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.478630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:47011 (local address 127.2.205.130:41895)
0929 03:02:05.478860 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.478868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.479417 (+ 549us) server_negotiation.cc:407] Connection header received
0929 03:02:05.479634 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.479639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.479724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.479803 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.480594 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.481659 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.484053 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.484895 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.485056 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.485132 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.485450 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.485588 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:02:05.485999 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.478491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41895 (local address 127.2.205.129:47011)
0929 03:02:05.479331 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.479346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.479504 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.479808 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.479811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.479829 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.480409 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.480420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.481770 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.481778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.483895 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.483913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.485152 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.485159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.485303 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.485326 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.485403 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.485573 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":112,"thread_start_us":515,"threads_started":1}
I20260929 03:02:05.486246 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.478693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:51443 (local address 127.2.205.131:43533)
0929 03:02:05.478886 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.478891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.479358 (+ 467us) server_negotiation.cc:407] Connection header received
0929 03:02:05.479538 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.479542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.479639 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.479743 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.480496 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.481598 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.484085 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.484884 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.485035 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.485114 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.485459 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.485598 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:02:05.486059 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.478491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.2.205.129:51443)
0929 03:02:05.479215 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.479234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.479408 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.479777 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.479782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.479809 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.480336 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.480353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.481720 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.481726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.483928 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.483942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.485111 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.485118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.485271 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.485320 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.485406 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.485582 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":175,"thread_start_us":391,"threads_started":1}
I20260929 03:02:05.487262 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4942430c840f7a86e4475a6cd48bb" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e" is_pre_election: true
I20260929 03:02:05.487322 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4942430c840f7a86e4475a6cd48bb" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" is_pre_election: true
I20260929 03:02:05.487748 10770 raft_consensus.cc:2454] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 0.
I20260929 03:02:05.487752 10909 raft_consensus.cc:2454] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 0.
I20260929 03:02:05.488319 10562 leader_election.cc:302] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, b661baa981ea47f5a3eb1589b99d7c4f; no voters:
I20260929 03:02:05.488675 10990 raft_consensus.cc:2781] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:05.488840 10990 raft_consensus.cc:482] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:05.488931 10990 raft_consensus.cc:3037] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.491076 10990 raft_consensus.cc:504] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.491698 10990 leader_election.cc:288] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 election: Requested vote from peers ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:05.492201 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4942430c840f7a86e4475a6cd48bb" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
I20260929 03:02:05.492251 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4942430c840f7a86e4475a6cd48bb" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e"
I20260929 03:02:05.492389 10909 raft_consensus.cc:3037] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.492493 10770 raft_consensus.cc:3037] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.494597 10909 raft_consensus.cc:2454] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 1.
I20260929 03:02:05.494663 10770 raft_consensus.cc:2454] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 1.
I20260929 03:02:05.495023 10563 leader_election.cc:302] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b661baa981ea47f5a3eb1589b99d7c4f, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:05.495337 10990 raft_consensus.cc:2781] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:05.495749 10990 raft_consensus.cc:686] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 1 LEADER]: Becoming Leader. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Running, Role: LEADER
I20260929 03:02:05.496403 10990 consensus_queue.cc:237] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } }
I20260929 03:02:05.500945 10474 catalog_manager.cc:5495] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129). New cstate: current_term: 1 leader_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } health_report { overall_health: UNKNOWN } } }
I20260929 03:02:05.606364 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.600552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.2.205.129:37135)
0929 03:02:05.600996 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.601000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.601400 (+ 400us) server_negotiation.cc:407] Connection header received
0929 03:02:05.601577 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.601581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.601663 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.601811 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:05.602201 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.602901 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.604288 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.604739 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.604894 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.605246 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.605854 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.605968 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.606050 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.606125 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260929 03:02:05.614986 10909 raft_consensus.cc:1261] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Refusing update from remote peer b661baa981ea47f5a3eb1589b99d7c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:05.614991 10770 raft_consensus.cc:1261] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Refusing update from remote peer b661baa981ea47f5a3eb1589b99d7c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:05.616125 10995 consensus_queue.cc:1035] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:05.616532 10990 consensus_queue.cc:1035] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:05.628731 11000 mvcc.cc:205] Tried to move back new op lower bound from 7334506191307280384 to 7334506190837432320. Current Snapshot: MvccSnapshot[applied={T|T < 7334506191307280384}]
I20260929 03:02:05.630115 11003 mvcc.cc:205] Tried to move back new op lower bound from 7334506191307280384 to 7334506190837432320. Current Snapshot: MvccSnapshot[applied={T|T < 7334506191307280384}]
I20260929 03:02:05.642871 10472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:05.643133 10472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:05.646551 10472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Sending DeleteTablet for 3 replicas of tablet dec4942430c840f7a86e4475a6cd48bb
I20260929 03:02:05.647624 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4942430c840f7a86e4475a6cd48bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:05 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46044
I20260929 03:02:05.647737 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4942430c840f7a86e4475a6cd48bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:05 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36886
I20260929 03:02:05.648324 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4942430c840f7a86e4475a6cd48bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:05 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45662
I20260929 03:02:05.649617 10472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:02:05.650315 10472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:05.657773 10888 tablet_service.cc:1467] Processing CreateTablet for tablet 8818d5a9913343a3bc36f14e5f45508c (DEFAULT_TABLE table=test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.658100 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 8818d5a9913343a3bc36f14e5f45508c (DEFAULT_TABLE table=test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.658308 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818d5a9913343a3bc36f14e5f45508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.658557 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818d5a9913343a3bc36f14e5f45508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.661262 10610 tablet_service.cc:1467] Processing CreateTablet for tablet 8818d5a9913343a3bc36f14e5f45508c (DEFAULT_TABLE table=test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:05.661774 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8818d5a9913343a3bc36f14e5f45508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.694327 11016 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Bootstrap starting.
I20260929 03:02:05.696914 11016 tablet_bootstrap.cc:654] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.698333 11017 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Bootstrap starting.
I20260929 03:02:05.700381 11017 tablet_bootstrap.cc:654] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.702107 11015 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Bootstrap starting.
I20260929 03:02:05.705574 11015 tablet_bootstrap.cc:654] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.706290 11014 tablet_replica.cc:330] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: stopping tablet replica
I20260929 03:02:05.706895 11012 tablet_replica.cc:330] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: stopping tablet replica
I20260929 03:02:05.707564 11012 raft_consensus.cc:2229] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:05.708599 11012 raft_consensus.cc:2258] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:05.710968 11012 ts_tablet_manager.cc:1905] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:05.714329 11017 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: No bootstrap required, opened a new log
I20260929 03:02:05.714529 11017 ts_tablet_manager.cc:1397] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20260929 03:02:05.715508 11017 raft_consensus.cc:348] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.715817 11017 raft_consensus.cc:374] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.715915 11017 raft_consensus.cc:729] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.716604 11014 raft_consensus.cc:2229] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:05.717387 11014 raft_consensus.cc:2258] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:05.719214 11017 consensus_queue.cc:260] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.720173 11013 tablet_replica.cc:330] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: stopping tablet replica
I20260929 03:02:05.720698 11013 raft_consensus.cc:2229] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:05.721621 11016 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: No bootstrap required, opened a new log
I20260929 03:02:05.721796 11016 ts_tablet_manager.cc:1397] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20260929 03:02:05.722436 11014 ts_tablet_manager.cc:1905] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:05.722985 11017 ts_tablet_manager.cc:1428] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent starting tablet: real 0.008s user 0.000s sys 0.004s
I20260929 03:02:05.723346 11013 raft_consensus.cc:2258] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:05.724025 11016 raft_consensus.cc:348] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.724270 11016 raft_consensus.cc:374] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.724364 11016 raft_consensus.cc:729] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.724611 11016 consensus_queue.cc:260] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.723381 11015 tablet_bootstrap.cc:492] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: No bootstrap required, opened a new log
I20260929 03:02:05.725112 11015 ts_tablet_manager.cc:1397] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.015s
I20260929 03:02:05.725181 11016 ts_tablet_manager.cc:1428] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:05.726608 11013 ts_tablet_manager.cc:1905] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:05.727505 11015 raft_consensus.cc:348] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.727830 11015 raft_consensus.cc:374] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.727957 11012 ts_tablet_manager.cc:1918] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:05.727988 11015 raft_consensus.cc:729] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.728113 11012 log.cc:1196] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/dec4942430c840f7a86e4475a6cd48bb
I20260929 03:02:05.728614 11012 ts_tablet_manager.cc:1939] T dec4942430c840f7a86e4475a6cd48bb P b661baa981ea47f5a3eb1589b99d7c4f: Deleting consensus metadata
I20260929 03:02:05.728890 11015 consensus_queue.cc:260] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.730355 10458 catalog_manager.cc:4841] TS b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135): tablet dec4942430c840f7a86e4475a6cd48bb (table test-table [id=29db7f9669cb493d889550adf7b3290c]) successfully deleted
I20260929 03:02:05.730552 11015 ts_tablet_manager.cc:1428] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 03:02:05.730938 11014 ts_tablet_manager.cc:1918] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:05.731139 11014 log.cc:1196] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/dec4942430c840f7a86e4475a6cd48bb
I20260929 03:02:05.731710 11014 ts_tablet_manager.cc:1939] T dec4942430c840f7a86e4475a6cd48bb P 91de360d29344e4fa720f0279162ba3e: Deleting consensus metadata
I20260929 03:02:05.733530 10459 catalog_manager.cc:4841] TS 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895): tablet dec4942430c840f7a86e4475a6cd48bb (table test-table [id=29db7f9669cb493d889550adf7b3290c]) successfully deleted
I20260929 03:02:05.735910 11013 ts_tablet_manager.cc:1918] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:05.736100 11013 log.cc:1196] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/dec4942430c840f7a86e4475a6cd48bb
I20260929 03:02:05.736624 11013 ts_tablet_manager.cc:1939] T dec4942430c840f7a86e4475a6cd48bb P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting consensus metadata
I20260929 03:02:05.738493 10460 catalog_manager.cc:4841] TS ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533): tablet dec4942430c840f7a86e4475a6cd48bb (table test-table [id=29db7f9669cb493d889550adf7b3290c]) successfully deleted
I20260929 03:02:05.742838 11002 raft_consensus.cc:482] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:05.743064 11002 raft_consensus.cc:504] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.744609 11002 leader_election.cc:288] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533)
I20260929 03:02:05.752346 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.745081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:48493 (local address 127.2.205.131:43533)
0929 03:02:05.745278 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.745284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.746603 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 03:02:05.746776 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.746781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.746854 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.746949 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.747564 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.748539 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.750314 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.751396 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.751563 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.751644 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.751951 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.752080 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:02:05.752503 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.745610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.2.205.130:48493)
0929 03:02:05.746492 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.746509 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.746649 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.746985 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.746990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.747018 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.747388 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.747399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.748652 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.748656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.750168 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.750180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.751595 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.751601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.751782 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.751818 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.751913 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.752054 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":1,"spinlock_wait_cycles":3840,"thread_start_us":545,"threads_started":1}
I20260929 03:02:05.752986 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.745255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:50047 (local address 127.2.205.129:37135)
0929 03:02:05.745427 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.745438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.746020 (+ 582us) server_negotiation.cc:407] Connection header received
0929 03:02:05.746265 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.746271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.746350 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.746418 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.747165 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.748204 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.750655 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.751839 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.752000 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.752066 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.752589 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.752713 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:02:05.753355 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818d5a9913343a3bc36f14e5f45508c" candidate_uuid: "91de360d29344e4fa720f0279162ba3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" is_pre_election: true
I20260929 03:02:05.753628 10909 raft_consensus.cc:2454] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de360d29344e4fa720f0279162ba3e in term 0.
I20260929 03:02:05.753679 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.745020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37135 (local address 127.2.205.130:50047)
0929 03:02:05.745887 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.745908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.746070 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.746602 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.746606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.746651 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.746973 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.746985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.748334 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.748340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.750483 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.750497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.752223 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.752229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.752408 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.752440 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.753172 (+ 732us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.753304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"spinlock_wait_cycles":3840,"thread_start_us":559,"threads_started":1}
I20260929 03:02:05.754130 10703 leader_election.cc:302] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:05.754536 11002 raft_consensus.cc:2781] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:05.754640 11002 raft_consensus.cc:482] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:05.754731 11002 raft_consensus.cc:3037] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.754659 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818d5a9913343a3bc36f14e5f45508c" candidate_uuid: "91de360d29344e4fa720f0279162ba3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" is_pre_election: true
I20260929 03:02:05.755129 10631 raft_consensus.cc:2454] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91de360d29344e4fa720f0279162ba3e in term 0.
I20260929 03:02:05.757082 11002 raft_consensus.cc:504] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.757628 11002 leader_election.cc:288] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [CANDIDATE]: Term 1 election: Requested vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533)
I20260929 03:02:05.758049 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818d5a9913343a3bc36f14e5f45508c" candidate_uuid: "91de360d29344e4fa720f0279162ba3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
I20260929 03:02:05.758121 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8818d5a9913343a3bc36f14e5f45508c" candidate_uuid: "91de360d29344e4fa720f0279162ba3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
I20260929 03:02:05.758242 10631 raft_consensus.cc:3037] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.758301 10909 raft_consensus.cc:3037] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.760221 10909 raft_consensus.cc:2454] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de360d29344e4fa720f0279162ba3e in term 1.
I20260929 03:02:05.760715 10703 leader_election.cc:302] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:05.761077 11002 raft_consensus.cc:2781] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:05.761431 10631 raft_consensus.cc:2454] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91de360d29344e4fa720f0279162ba3e in term 1.
I20260929 03:02:05.761670 11002 raft_consensus.cc:686] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 1 LEADER]: Becoming Leader. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Running, Role: LEADER
I20260929 03:02:05.762267 11002 consensus_queue.cc:237] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.765473 10472 catalog_manager.cc:5495] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 91de360d29344e4fa720f0279162ba3e (127.2.205.130). New cstate: current_term: 1 leader_uuid: "91de360d29344e4fa720f0279162ba3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:02:05 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650922, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.190@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:05.817752 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.803381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38399 (local address 127.0.0.1:35938)
0929 03:02:05.804178 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.804193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.804350 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.804655 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.804663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.805439 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:05.805687 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.805698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.806797 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.806801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.807783 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.807796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.808572 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.808578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.808774 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.809523 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:05.809543 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:05.812061 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:05.815192 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:05.815208 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:05.815218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:05.815681 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:05.816029 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:05.816034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:05.816038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:05.816175 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:05.816687 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:05.816696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:05.816908 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.817063 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.817376 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":557,"threads_started":1}
I20260929 03:02:05.818053 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.803365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.2.205.190:38399)
0929 03:02:05.804039 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.804045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.804287 (+ 242us) server_negotiation.cc:407] Connection header received
0929 03:02:05.804462 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.804466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.804523 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.804611 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:05.805862 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.806648 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.807934 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.808392 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.808548 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.812260 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:05.812265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:05.812268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:05.812308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:05.815059 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:05.815792 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:05.815797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:05.815800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:05.815928 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:05.816269 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:05.816274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:05.816277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:05.816554 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:05.816686 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.817260 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.817541 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":494,"thread_start_us":321,"threads_started":1}
I20260929 03:02:05.823760 10472 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:35938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:02:05.825119 10472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:05.831866 10611 tablet_service.cc:1467] Processing CreateTablet for tablet 6c39ab67bd1743c49af722fde03ad2c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ac5db5fc96641d0baeefc8d605e982a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:02:05.832010 10750 tablet_service.cc:1467] Processing CreateTablet for tablet 6c39ab67bd1743c49af722fde03ad2c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ac5db5fc96641d0baeefc8d605e982a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:02:05.832209 10889 tablet_service.cc:1467] Processing CreateTablet for tablet 6c39ab67bd1743c49af722fde03ad2c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ac5db5fc96641d0baeefc8d605e982a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:02:05.832499 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c39ab67bd1743c49af722fde03ad2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.832584 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c39ab67bd1743c49af722fde03ad2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.832726 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c39ab67bd1743c49af722fde03ad2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:05.839877 11016 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: Bootstrap starting.
I20260929 03:02:05.840109 11015 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: Bootstrap starting.
I20260929 03:02:05.840256 11017 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Bootstrap starting.
I20260929 03:02:05.841524 11016 tablet_bootstrap.cc:654] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.841794 11017 tablet_bootstrap.cc:654] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.841794 11015 tablet_bootstrap.cc:654] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:05.853514 11017 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: No bootstrap required, opened a new log
I20260929 03:02:05.853688 11017 ts_tablet_manager.cc:1397] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260929 03:02:05.853734 11015 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: No bootstrap required, opened a new log
I20260929 03:02:05.853885 11015 ts_tablet_manager.cc:1397] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260929 03:02:05.854450 11017 raft_consensus.cc:348] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.854534 11015 raft_consensus.cc:348] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.854734 11017 raft_consensus.cc:374] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.854775 11015 raft_consensus.cc:374] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.854815 11017 raft_consensus.cc:729] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.854862 11015 raft_consensus.cc:729] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.855053 11017 consensus_queue.cc:260] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.855145 11015 consensus_queue.cc:260] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.855609 11017 ts_tablet_manager.cc:1428] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 03:02:05.855507 11002 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.855773 11002 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.855605 11001 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.856910 11001 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.857988 11015 ts_tablet_manager.cc:1428] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:02:05.859628 11016 tablet_bootstrap.cc:492] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: No bootstrap required, opened a new log
I20260929 03:02:05.859846 11016 ts_tablet_manager.cc:1397] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.015s
I20260929 03:02:05.859905 11001 raft_consensus.cc:482] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:05.860059 11001 raft_consensus.cc:504] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.860575 11016 raft_consensus.cc:348] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.860918 11016 raft_consensus.cc:374] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:05.861007 11016 raft_consensus.cc:729] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Initialized, Role: FOLLOWER
I20260929 03:02:05.861460 11016 consensus_queue.cc:260] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.861735 11001 leader_election.cc:288] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:05.864552 10990 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.864830 10990 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.867167 11016 ts_tablet_manager.cc:1428] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260929 03:02:05.877177 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.862180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:42305 (local address 127.2.205.130:41895)
0929 03:02:05.862413 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.862419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.870811 (+ 8392us) server_negotiation.cc:407] Connection header received
0929 03:02:05.871031 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.871036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.871103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.871193 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.871959 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.873081 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.875464 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.876218 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.876379 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.876453 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.876765 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.876912 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:02:05.877413 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.862489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:60763 (local address 127.2.205.129:37135)
0929 03:02:05.862642 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.862648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.870255 (+ 7607us) server_negotiation.cc:407] Connection header received
0929 03:02:05.870417 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.870422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.870535 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.870617 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:05.871489 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.872471 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.875232 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.876459 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.876617 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.876690 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.876992 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.877123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:02:05.877554 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.862099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37135 (local address 127.2.205.131:60763)
0929 03:02:05.870113 (+ 8014us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.870133 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.870265 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.870798 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.870803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.870862 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.871273 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.871348 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.872593 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.872598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.875081 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.875095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.876665 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.876671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.876838 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.876870 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.876951 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.877073 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7889,"mutex_wait_us":18,"spinlock_wait_cycles":44032,"thread_start_us":8390,"threads_started":1}
I20260929 03:02:05.877880 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.862554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41895 (local address 127.2.205.131:42305)
0929 03:02:05.870716 (+ 8162us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.870732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.870865 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.871224 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.871232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.871259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:05.871650 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.871661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.873195 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.873200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.875340 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.875349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.876393 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.876399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.876564 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.876598 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.877344 (+ 746us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.877471 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8056,"mutex_wait_us":18,"spinlock_wait_cycles":1792,"thread_start_us":7289,"threads_started":1}
I20260929 03:02:05.878365 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c39ab67bd1743c49af722fde03ad2c1" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" is_pre_election: true
I20260929 03:02:05.878468 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c39ab67bd1743c49af722fde03ad2c1" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e" is_pre_election: true
I20260929 03:02:05.878659 10631 raft_consensus.cc:2454] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 0.
I20260929 03:02:05.878696 10770 raft_consensus.cc:2454] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 0.
I20260929 03:02:05.879195 10840 leader_election.cc:302] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b661baa981ea47f5a3eb1589b99d7c4f, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:05.879660 11001 raft_consensus.cc:2781] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:05.879786 11001 raft_consensus.cc:482] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:05.879879 11001 raft_consensus.cc:3037] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.882975 11001 raft_consensus.cc:504] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.883673 11001 leader_election.cc:288] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 election: Requested vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:05.884224 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c39ab67bd1743c49af722fde03ad2c1" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e"
I20260929 03:02:05.884454 10770 raft_consensus.cc:3037] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.884990 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c39ab67bd1743c49af722fde03ad2c1" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
I20260929 03:02:05.885232 10631 raft_consensus.cc:3037] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:05.887415 10770 raft_consensus.cc:2454] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 1.
I20260929 03:02:05.887980 10841 leader_election.cc:302] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:05.888101 10631 raft_consensus.cc:2454] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 1.
I20260929 03:02:05.888381 11001 raft_consensus.cc:2781] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:05.888937 11001 raft_consensus.cc:686] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 LEADER]: Becoming Leader. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Running, Role: LEADER
I20260929 03:02:05.889351 11001 consensus_queue.cc:237] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:05.891633 11037 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: TxnStatusTablet state changed. Reason: New leader ed6ee7ce65474dcebd4fd63fc31cd2e5. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.892110 11037 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:02:05.892967 11039 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:02:05.893195 11039 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:02:05.893646 10472 catalog_manager.cc:5495] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131). New cstate: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260929 03:02:05.918099 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.912395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.2.205.131:43533)
0929 03:02:05.912578 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.912584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.912852 (+ 268us) server_negotiation.cc:407] Connection header received
0929 03:02:05.913173 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.913182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.913260 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.913376 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:05.913736 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.914528 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.915778 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.916230 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.916342 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.916832 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.917430 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.917603 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.917686 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.917769 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:02:05.918183 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.912401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.0.0.1:36896)
0929 03:02:05.912775 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.912794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.912960 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.913419 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.913424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.913436 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:02:05.913609 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.913618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.914647 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.914653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.915636 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.915648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.916385 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.916392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.916569 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.916623 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.917561 (+ 938us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.917581 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.917648 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:02:05.917801 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 03:02:05.922142 10770 raft_consensus.cc:1261] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Refusing update from remote peer ed6ee7ce65474dcebd4fd63fc31cd2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:05.922327 10631 raft_consensus.cc:1261] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Refusing update from remote peer ed6ee7ce65474dcebd4fd63fc31cd2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:05.922971 11001 consensus_queue.cc:1035] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:05.923664 11001 consensus_queue.cc:1035] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:05.926847 10990 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: TxnStatusTablet state changed. Reason: New leader ed6ee7ce65474dcebd4fd63fc31cd2e5. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.927146 10990 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.928952 11002 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: TxnStatusTablet state changed. Reason: New leader ed6ee7ce65474dcebd4fd63fc31cd2e5. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.928969 11001 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.929234 11001 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:02:05.929242 11002 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.933233 11001 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.933526 11001 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P ed6ee7ce65474dcebd4fd63fc31cd2e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:02:05.936069 11004 mvcc.cc:205] Tried to move back new op lower bound from 7334506192570093568 to 7334506192447369216. Current Snapshot: MvccSnapshot[applied={T|T < 7334506192570093568}]
I20260929 03:02:05.936398 11002 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.936688 11002 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P 91de360d29344e4fa720f0279162ba3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.938468 10990 tablet_replica.cc:439] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } } }
I20260929 03:02:05.938705 10990 tablet_replica.cc:442] T 6c39ab67bd1743c49af722fde03ad2c1 P b661baa981ea47f5a3eb1589b99d7c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:02:05.953238 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.944877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.2.205.130:41895)
0929 03:02:05.945037 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.945042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.945061 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:02:05.945297 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.945303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.945413 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.945541 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:05.946056 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.947174 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.949168 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.949798 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.949908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.950480 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.951144 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.951262 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:05.952862 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0929 03:02:05.952954 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:02:06.000435 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.994738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:52789 (local address 127.2.205.131:43533)
0929 03:02:05.994966 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:05.994972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:05.995635 (+ 663us) server_negotiation.cc:407] Connection header received
0929 03:02:05.995834 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:05.995838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:05.995903 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:05.995983 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:05.996478 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.997260 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.998625 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.999144 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.999260 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.999694 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.999799 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.999870 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.000019 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.000102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:02:06.000502 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:05.994769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.2.205.130:52789)
0929 03:02:05.995516 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:05.995535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:05.995698 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:05.996020 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:05.996023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:05.996035 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:02:05.996314 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.996322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.997398 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.997403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.998483 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:05.998496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:05.999321 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:05.999328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:05.999470 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:05.999530 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:02:05.999890 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:02:05.999902 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:02:05.999981 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:02:06.000132 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":382,"threads_started":1}
I20260929 03:02:06.013310 10909 raft_consensus.cc:1261] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Refusing update from remote peer 91de360d29344e4fa720f0279162ba3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:06.013397 10631 raft_consensus.cc:1261] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Refusing update from remote peer 91de360d29344e4fa720f0279162ba3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:06.014257 11002 consensus_queue.cc:1035] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:06.014925 11002 consensus_queue.cc:1035] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:06.075865 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.062250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:46099 (local address 127.2.205.130:41895)
0929 03:02:06.062437 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.062443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.069369 (+ 6926us) server_negotiation.cc:407] Connection header received
0929 03:02:06.069696 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.069703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.069792 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.069884 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:06.070849 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.071797 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.073661 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.074362 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.074464 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.075160 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.075269 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.075339 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.075479 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.075557 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:02:06.075986 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.062604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41895 (local address 127.2.205.131:46099)
0929 03:02:06.069257 (+ 6653us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:06.069278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:06.069495 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:06.070053 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:06.070058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:06.070072 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:02:06.070610 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:06.070628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.071936 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.071948 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:06.073502 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:06.073518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.074704 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.074715 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:06.074924 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.074992 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.075360 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.075371 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:02:06.075440 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:02:06.075572 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6442,"thread_start_us":6325,"threads_started":1}
I20260929 03:02:06.117585 10472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:06.117899 10472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:06.121940 10472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Sending DeleteTablet for 3 replicas of tablet 8818d5a9913343a3bc36f14e5f45508c
I20260929 03:02:06.123111 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet 8818d5a9913343a3bc36f14e5f45508c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46044
I20260929 03:02:06.123322 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8818d5a9913343a3bc36f14e5f45508c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36886
I20260929 03:02:06.123414 11012 tablet_replica.cc:330] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: stopping tablet replica
I20260929 03:02:06.123510 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8818d5a9913343a3bc36f14e5f45508c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45662
I20260929 03:02:06.123616 11013 tablet_replica.cc:330] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: stopping tablet replica
I20260929 03:02:06.123611 11012 raft_consensus.cc:2229] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:06.123783 11014 tablet_replica.cc:330] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: stopping tablet replica
I20260929 03:02:06.123786 11013 raft_consensus.cc:2229] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:06.123805 11012 raft_consensus.cc:2258] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.123963 11013 raft_consensus.cc:2258] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.123981 11014 raft_consensus.cc:2229] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:06.124508 11014 raft_consensus.cc:2258] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.125135 11013 ts_tablet_manager.cc:1905] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.125135 11012 ts_tablet_manager.cc:1905] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.127302 11014 ts_tablet_manager.cc:1905] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.132342 11012 ts_tablet_manager.cc:1918] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:06.132342 11013 ts_tablet_manager.cc:1918] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:06.132550 11012 log.cc:1196] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/8818d5a9913343a3bc36f14e5f45508c
I20260929 03:02:06.132712 11013 log.cc:1196] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/8818d5a9913343a3bc36f14e5f45508c
I20260929 03:02:06.133033 11012 ts_tablet_manager.cc:1939] T 8818d5a9913343a3bc36f14e5f45508c P b661baa981ea47f5a3eb1589b99d7c4f: Deleting consensus metadata
I20260929 03:02:06.133616 11013 ts_tablet_manager.cc:1939] T 8818d5a9913343a3bc36f14e5f45508c P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting consensus metadata
I20260929 03:02:06.134266 10458 catalog_manager.cc:4841] TS b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135): tablet 8818d5a9913343a3bc36f14e5f45508c (table test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]) successfully deleted
I20260929 03:02:06.135532 10460 catalog_manager.cc:4841] TS ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533): tablet 8818d5a9913343a3bc36f14e5f45508c (table test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]) successfully deleted
I20260929 03:02:06.147209 11014 ts_tablet_manager.cc:1918] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:06.147550 11014 log.cc:1196] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/8818d5a9913343a3bc36f14e5f45508c
I20260929 03:02:06.148114 11014 ts_tablet_manager.cc:1939] T 8818d5a9913343a3bc36f14e5f45508c P 91de360d29344e4fa720f0279162ba3e: Deleting consensus metadata
I20260929 03:02:06.149549 10459 catalog_manager.cc:4841] TS 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895): tablet 8818d5a9913343a3bc36f14e5f45508c (table test-table [id=9a6c53a8b4e3405f893de413a3b1a14f]) successfully deleted
Sep 29 03:02:06 dist-test-slave-530n krb5kdc[10429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:02:06 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:06 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:06.256397 11030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:06.231777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.2.205.190:38399)
0929 03:02:06.232028 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.232035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.242481 (+ 10446us) server_negotiation.cc:407] Connection header received
0929 03:02:06.243029 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.243037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.243128 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.243206 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:06.245268 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.246412 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.248335 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.249018 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.249118 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.256159 (+ 7041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35950: BlockingRecv error: recv got EOF from 127.0.0.1:35950 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 03:02:06 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM
I20260929 03:02:06.277885 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.260239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.2.205.190:38399)
0929 03:02:06.260403 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.260409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.260951 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:02:06.261195 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.261200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.261281 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.261378 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:06.262782 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.263827 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.265695 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.266380 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.266508 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.271417 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:06.271425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:06.271429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:06.271473 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:06.275044 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:06.275796 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:06.275802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:06.275805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:06.275979 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:06.276355 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:06.276361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:06.276365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:06.276704 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:06.276998 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.277281 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.277518 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:02:06.282373 10472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:02:06.283102 10472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:06.291350 10611 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5bf525e24446908314c00e2f019634 (DEFAULT_TABLE table=test-table [id=85e0d98cb5174a81879224915309ac64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.291646 10889 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5bf525e24446908314c00e2f019634 (DEFAULT_TABLE table=test-table [id=85e0d98cb5174a81879224915309ac64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.291849 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5bf525e24446908314c00e2f019634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.292129 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5bf525e24446908314c00e2f019634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.298949 11016 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Bootstrap starting.
I20260929 03:02:06.299820 10750 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5bf525e24446908314c00e2f019634 (DEFAULT_TABLE table=test-table [id=85e0d98cb5174a81879224915309ac64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.300279 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5bf525e24446908314c00e2f019634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.300467 11016 tablet_bootstrap.cc:654] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.310140 11017 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Bootstrap starting.
I20260929 03:02:06.311997 11017 tablet_bootstrap.cc:654] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.318701 11015 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Bootstrap starting.
I20260929 03:02:06.320591 11015 tablet_bootstrap.cc:654] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.343935 11015 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: No bootstrap required, opened a new log
I20260929 03:02:06.344225 11015 ts_tablet_manager.cc:1397] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.003s
I20260929 03:02:06.345145 11015 raft_consensus.cc:348] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.345932 11016 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: No bootstrap required, opened a new log
I20260929 03:02:06.346086 11015 raft_consensus.cc:374] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.346139 11016 ts_tablet_manager.cc:1397] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.017s
I20260929 03:02:06.346251 11015 raft_consensus.cc:729] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.346627 11017 tablet_bootstrap.cc:492] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: No bootstrap required, opened a new log
I20260929 03:02:06.346783 11017 ts_tablet_manager.cc:1397] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20260929 03:02:06.347229 11016 raft_consensus.cc:348] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.347559 11016 raft_consensus.cc:374] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.347556 11017 raft_consensus.cc:348] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.347702 11016 raft_consensus.cc:729] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.347810 11017 raft_consensus.cc:374] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.347913 11017 raft_consensus.cc:729] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.348119 11016 consensus_queue.cc:260] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.348201 11015 consensus_queue.cc:260] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.350481 11015 ts_tablet_manager.cc:1428] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:02:06.350801 11017 consensus_queue.cc:260] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.352298 11017 ts_tablet_manager.cc:1428] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:02:06.352923 11016 ts_tablet_manager.cc:1428] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 03:02:06.465982 11051 raft_consensus.cc:482] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:06.466224 11051 raft_consensus.cc:504] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.466979 11051 leader_election.cc:288] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:06.467916 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5bf525e24446908314c00e2f019634" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" is_pre_election: true
I20260929 03:02:06.468183 10631 raft_consensus.cc:2454] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 0.
I20260929 03:02:06.469051 10840 leader_election.cc:302] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b661baa981ea47f5a3eb1589b99d7c4f, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:06.469810 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5bf525e24446908314c00e2f019634" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e" is_pre_election: true
I20260929 03:02:06.470115 10770 raft_consensus.cc:2454] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 0.
I20260929 03:02:06.470990 11051 raft_consensus.cc:2781] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:06.471103 11051 raft_consensus.cc:482] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:06.471191 11051 raft_consensus.cc:3037] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.475286 11051 raft_consensus.cc:504] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.476451 10631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5bf525e24446908314c00e2f019634" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661baa981ea47f5a3eb1589b99d7c4f"
I20260929 03:02:06.476671 10631 raft_consensus.cc:3037] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.478266 11051 leader_election.cc:288] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 election: Requested vote from peers b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135), 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895)
I20260929 03:02:06.479264 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5bf525e24446908314c00e2f019634" candidate_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e"
I20260929 03:02:06.479486 10770 raft_consensus.cc:3037] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.479702 10631 raft_consensus.cc:2454] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 1.
I20260929 03:02:06.480192 10840 leader_election.cc:302] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b661baa981ea47f5a3eb1589b99d7c4f, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:06.480482 11051 raft_consensus.cc:2781] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:06.480633 11051 raft_consensus.cc:686] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 LEADER]: Becoming Leader. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Running, Role: LEADER
I20260929 03:02:06.481045 11051 consensus_queue.cc:237] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.482471 10770 raft_consensus.cc:2454] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ee7ce65474dcebd4fd63fc31cd2e5 in term 1.
I20260929 03:02:06.483958 10472 catalog_manager.cc:5495] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131). New cstate: current_term: 1 leader_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260929 03:02:06.528075 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.521814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.2.205.131:43533)
0929 03:02:06.522434 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.522441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.522463 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:02:06.522539 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.522544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.522642 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.522764 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:06.523131 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.524336 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.525828 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.526500 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.526653 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.527143 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.527295 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.527402 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.527624 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.527717 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":365,"threads_started":1}
I20260929 03:02:06.531733 10770 raft_consensus.cc:1261] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Refusing update from remote peer ed6ee7ce65474dcebd4fd63fc31cd2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:06.531752 10631 raft_consensus.cc:1261] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Refusing update from remote peer ed6ee7ce65474dcebd4fd63fc31cd2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:06.532375 11047 consensus_queue.cc:1035] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:06.532706 11051 consensus_queue.cc:1035] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:06.549428 10472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:06.549667 10472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:06.552384 10472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Sending DeleteTablet for 3 replicas of tablet 9c5bf525e24446908314c00e2f019634
I20260929 03:02:06.553288 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5bf525e24446908314c00e2f019634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36886
I20260929 03:02:06.553618 11013 tablet_replica.cc:330] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: stopping tablet replica
I20260929 03:02:06.553803 11013 raft_consensus.cc:2229] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:06.554317 11013 raft_consensus.cc:2258] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.555145 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5bf525e24446908314c00e2f019634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46044
I20260929 03:02:06.555410 11012 tablet_replica.cc:330] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: stopping tablet replica
I20260929 03:02:06.555576 11012 raft_consensus.cc:2229] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:06.555787 11012 raft_consensus.cc:2258] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.556413 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5bf525e24446908314c00e2f019634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:06 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45662
I20260929 03:02:06.556732 11014 tablet_replica.cc:330] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: stopping tablet replica
I20260929 03:02:06.556908 11013 ts_tablet_manager.cc:1905] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.556932 11014 raft_consensus.cc:2229] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:06.557152 11014 raft_consensus.cc:2258] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:06.558176 11014 ts_tablet_manager.cc:1905] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.563135 11013 ts_tablet_manager.cc:1918] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:06.563270 11013 log.cc:1196] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/9c5bf525e24446908314c00e2f019634
I20260929 03:02:06.563649 11013 ts_tablet_manager.cc:1939] T 9c5bf525e24446908314c00e2f019634 P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting consensus metadata
I20260929 03:02:06.564601 10460 catalog_manager.cc:4841] TS ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533): tablet 9c5bf525e24446908314c00e2f019634 (table test-table [id=85e0d98cb5174a81879224915309ac64]) successfully deleted
I20260929 03:02:06.564807 11014 ts_tablet_manager.cc:1918] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:06.564682 10472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:02:06.564944 11014 log.cc:1196] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/9c5bf525e24446908314c00e2f019634
I20260929 03:02:06.565302 11014 ts_tablet_manager.cc:1939] T 9c5bf525e24446908314c00e2f019634 P 91de360d29344e4fa720f0279162ba3e: Deleting consensus metadata
W20260929 03:02:06.565354 10472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:06.566334 10459 catalog_manager.cc:4841] TS 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895): tablet 9c5bf525e24446908314c00e2f019634 (table test-table [id=85e0d98cb5174a81879224915309ac64]) successfully deleted
I20260929 03:02:06.568037 11012 ts_tablet_manager.cc:1905] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:06.576283 10611 tablet_service.cc:1467] Processing CreateTablet for tablet e36f77f01be74f4eb62118c2e21b65ea (DEFAULT_TABLE table=test-table [id=bdb900eb6865442da15f57e4afe3c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.576553 10889 tablet_service.cc:1467] Processing CreateTablet for tablet e36f77f01be74f4eb62118c2e21b65ea (DEFAULT_TABLE table=test-table [id=bdb900eb6865442da15f57e4afe3c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.576777 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36f77f01be74f4eb62118c2e21b65ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.577015 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36f77f01be74f4eb62118c2e21b65ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.581919 10750 tablet_service.cc:1467] Processing CreateTablet for tablet e36f77f01be74f4eb62118c2e21b65ea (DEFAULT_TABLE table=test-table [id=bdb900eb6865442da15f57e4afe3c167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:06.582434 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e36f77f01be74f4eb62118c2e21b65ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:06.584416 11017 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Bootstrap starting.
I20260929 03:02:06.586233 11017 tablet_bootstrap.cc:654] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.591562 11016 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Bootstrap starting.
I20260929 03:02:06.593458 11016 tablet_bootstrap.cc:654] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.593689 11012 ts_tablet_manager.cc:1918] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:06.593847 11012 log.cc:1196] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/9c5bf525e24446908314c00e2f019634
I20260929 03:02:06.594378 11012 ts_tablet_manager.cc:1939] T 9c5bf525e24446908314c00e2f019634 P b661baa981ea47f5a3eb1589b99d7c4f: Deleting consensus metadata
I20260929 03:02:06.595578 11017 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: No bootstrap required, opened a new log
I20260929 03:02:06.595556 10458 catalog_manager.cc:4841] TS b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135): tablet 9c5bf525e24446908314c00e2f019634 (table test-table [id=85e0d98cb5174a81879224915309ac64]) successfully deleted
I20260929 03:02:06.595860 11017 ts_tablet_manager.cc:1397] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260929 03:02:06.597054 11017 raft_consensus.cc:348] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.597394 11017 raft_consensus.cc:374] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.597527 11017 raft_consensus.cc:729] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ee7ce65474dcebd4fd63fc31cd2e5, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.597550 11015 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Bootstrap starting.
I20260929 03:02:06.597870 11017 consensus_queue.cc:260] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.599139 11017 ts_tablet_manager.cc:1428] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:06.599434 11015 tablet_bootstrap.cc:654] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:06.601701 11015 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: No bootstrap required, opened a new log
I20260929 03:02:06.601882 11015 ts_tablet_manager.cc:1397] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 03:02:06.602632 11015 raft_consensus.cc:348] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.602890 11015 raft_consensus.cc:374] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.602973 11015 raft_consensus.cc:729] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91de360d29344e4fa720f0279162ba3e, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.603241 11015 consensus_queue.cc:260] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.604242 11015 ts_tablet_manager.cc:1428] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 03:02:06.610011 11016 tablet_bootstrap.cc:492] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: No bootstrap required, opened a new log
I20260929 03:02:06.610193 11016 ts_tablet_manager.cc:1397] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20260929 03:02:06.610961 11016 raft_consensus.cc:348] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.611214 11016 raft_consensus.cc:374] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:06.611284 11016 raft_consensus.cc:729] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Initialized, Role: FOLLOWER
I20260929 03:02:06.611506 11016 consensus_queue.cc:260] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.612037 11016 ts_tablet_manager.cc:1428] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 03:02:06.904573 10990 raft_consensus.cc:482] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:06.904817 10990 raft_consensus.cc:504] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.905514 10990 leader_election.cc:288] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895), ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533)
I20260929 03:02:06.906121 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36f77f01be74f4eb62118c2e21b65ea" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e" is_pre_election: true
I20260929 03:02:06.906193 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36f77f01be74f4eb62118c2e21b65ea" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" is_pre_election: true
I20260929 03:02:06.906392 10770 raft_consensus.cc:2454] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 0.
I20260929 03:02:06.906422 10909 raft_consensus.cc:2454] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 0.
I20260929 03:02:06.906936 10563 leader_election.cc:302] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, b661baa981ea47f5a3eb1589b99d7c4f, ed6ee7ce65474dcebd4fd63fc31cd2e5; no voters:
I20260929 03:02:06.907253 10990 raft_consensus.cc:2781] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:06.907398 10990 raft_consensus.cc:482] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:06.907517 10990 raft_consensus.cc:3037] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.910820 10990 raft_consensus.cc:504] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.911644 10990 leader_election.cc:288] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 election: Requested vote from peers 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895), ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533)
I20260929 03:02:06.912187 10770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36f77f01be74f4eb62118c2e21b65ea" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91de360d29344e4fa720f0279162ba3e"
I20260929 03:02:06.912204 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e36f77f01be74f4eb62118c2e21b65ea" candidate_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5"
I20260929 03:02:06.912395 10770 raft_consensus.cc:3037] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.912443 10909 raft_consensus.cc:3037] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:06.914790 10770 raft_consensus.cc:2454] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 1.
I20260929 03:02:06.915192 10562 leader_election.cc:302] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91de360d29344e4fa720f0279162ba3e, b661baa981ea47f5a3eb1589b99d7c4f; no voters:
I20260929 03:02:06.915452 10990 raft_consensus.cc:2781] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:06.915500 10909 raft_consensus.cc:2454] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661baa981ea47f5a3eb1589b99d7c4f in term 1.
I20260929 03:02:06.915827 10990 raft_consensus.cc:686] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 1 LEADER]: Becoming Leader. State: Replica: b661baa981ea47f5a3eb1589b99d7c4f, State: Running, Role: LEADER
I20260929 03:02:06.916141 10990 consensus_queue.cc:237] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } }
I20260929 03:02:06.919019 10472 catalog_manager.cc:5495] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129). New cstate: current_term: 1 leader_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b661baa981ea47f5a3eb1589b99d7c4f" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260929 03:02:06.985344 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.975730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.2.205.129:37135)
0929 03:02:06.980712 (+ 4982us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.980718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.980737 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:02:06.980838 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.980843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.980914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.981012 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:06.981452 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.982271 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.983590 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.984011 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.984117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.984565 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.984685 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.984785 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.984931 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.985031 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4850,"thread_start_us":4660,"threads_started":1}
I20260929 03:02:06.999330 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.994493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:36495 (local address 127.2.205.131:43533)
0929 03:02:06.994662 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:06.994667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:06.995188 (+ 521us) server_negotiation.cc:407] Connection header received
0929 03:02:06.995343 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:06.995347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:06.995415 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:06.995492 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:06.995879 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.996579 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.997926 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.998307 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.998404 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.998789 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.998891 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.998961 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:06.999058 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:02:06.999123 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:02:06.999420 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:06.994481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:43533 (local address 127.2.205.129:36495)
0929 03:02:06.995104 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:06.995120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:06.995241 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:06.995512 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:06.995515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:06.995524 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:02:06.995750 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:06.995758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.996681 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.996684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:06.997804 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:06.997817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:06.998459 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:06.998464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:06.998588 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:06.998644 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:02:06.998964 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:02:06.998977 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:02:06.999029 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:02:06.999138 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
I20260929 03:02:07.007519 10909 raft_consensus.cc:1261] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Refusing update from remote peer b661baa981ea47f5a3eb1589b99d7c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:07.007673 10770 raft_consensus.cc:1261] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Refusing update from remote peer b661baa981ea47f5a3eb1589b99d7c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:07.008131 11080 consensus_queue.cc:1035] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ee7ce65474dcebd4fd63fc31cd2e5" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:07.008500 10990 consensus_queue.cc:1035] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91de360d29344e4fa720f0279162ba3e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:07.036037 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:07.030766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:36863 (local address 127.2.205.129:37135)
0929 03:02:07.030917 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:07.030925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:07.031307 (+ 382us) server_negotiation.cc:407] Connection header received
0929 03:02:07.031488 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:07.031493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:07.031559 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:07.031627 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:07.032130 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.033015 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.034452 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.034873 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.034978 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:07.035354 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:07.035460 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:07.035539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:07.035663 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:02:07.035746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:02:07.036175 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:07.030732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37135 (local address 127.2.205.131:36863)
0929 03:02:07.031215 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:07.031232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:07.031352 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:07.031654 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:07.031658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:07.031671 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:02:07.031960 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:07.031977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.033139 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.033145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:07.034329 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:07.034341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.035042 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.035049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:07.035174 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:07.035214 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:02:07.035553 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:02:07.035565 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:02:07.035634 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:02:07.035775 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260929 03:02:07.064513 10474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:07.064785 10474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:07.067348 10474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ecff5ecf1964f3ba05212d4da427bab: Sending DeleteTablet for 3 replicas of tablet e36f77f01be74f4eb62118c2e21b65ea
I20260929 03:02:07.068099 10611 tablet_service.cc:1514] Processing DeleteTablet for tablet e36f77f01be74f4eb62118c2e21b65ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:07 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46044
I20260929 03:02:07.068219 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet e36f77f01be74f4eb62118c2e21b65ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:07 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36886
I20260929 03:02:07.068246 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet e36f77f01be74f4eb62118c2e21b65ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:07 UTC) from {username='oryx', principal='oryx/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45662
I20260929 03:02:07.068380 11012 tablet_replica.cc:330] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: stopping tablet replica
I20260929 03:02:07.068650 11012 raft_consensus.cc:2229] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:07.069016 11012 raft_consensus.cc:2258] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:07.069664 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10545
I20260929 03:02:07.069904 11092 tablet_replica.cc:330] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: stopping tablet replica
I20260929 03:02:07.070073 11093 tablet_replica.cc:330] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: stopping tablet replica
I20260929 03:02:07.070241 11093 raft_consensus.cc:2229] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:07.070340 11012 ts_tablet_manager.cc:1905] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:07.070149 11092 raft_consensus.cc:2229] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:07.070441 11093 raft_consensus.cc:2258] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:07.070596 11092 raft_consensus.cc:2258] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:07.072649 11092 ts_tablet_manager.cc:1905] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:07.072948 11093 ts_tablet_manager.cc:1905] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:07.077139 11012 ts_tablet_manager.cc:1918] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:07.077371 11012 log.cc:1196] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/e36f77f01be74f4eb62118c2e21b65ea
I20260929 03:02:07.077927 11012 ts_tablet_manager.cc:1939] T e36f77f01be74f4eb62118c2e21b65ea P b661baa981ea47f5a3eb1589b99d7c4f: Deleting consensus metadata
I20260929 03:02:07.079231 10458 catalog_manager.cc:4841] TS b661baa981ea47f5a3eb1589b99d7c4f (127.2.205.129:37135): tablet e36f77f01be74f4eb62118c2e21b65ea (table test-table [id=bdb900eb6865442da15f57e4afe3c167]) successfully deleted
I20260929 03:02:07.079367 11092 ts_tablet_manager.cc:1918] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:07.079494 11092 log.cc:1196] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/e36f77f01be74f4eb62118c2e21b65ea
I20260929 03:02:07.079674 11093 ts_tablet_manager.cc:1918] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:02:07.079886 11093 log.cc:1196] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/e36f77f01be74f4eb62118c2e21b65ea
I20260929 03:02:07.079941 11092 ts_tablet_manager.cc:1939] T e36f77f01be74f4eb62118c2e21b65ea P ed6ee7ce65474dcebd4fd63fc31cd2e5: Deleting consensus metadata
I20260929 03:02:07.080289 11093 ts_tablet_manager.cc:1939] T e36f77f01be74f4eb62118c2e21b65ea P 91de360d29344e4fa720f0279162ba3e: Deleting consensus metadata
I20260929 03:02:07.081195 10460 catalog_manager.cc:4841] TS ed6ee7ce65474dcebd4fd63fc31cd2e5 (127.2.205.131:43533): tablet e36f77f01be74f4eb62118c2e21b65ea (table test-table [id=bdb900eb6865442da15f57e4afe3c167]) successfully deleted
I20260929 03:02:07.081302 10459 catalog_manager.cc:4841] TS 91de360d29344e4fa720f0279162ba3e (127.2.205.130:41895): tablet e36f77f01be74f4eb62118c2e21b65ea (table test-table [id=bdb900eb6865442da15f57e4afe3c167]) successfully deleted
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.129@KRBTEST.COM
I20260929 03:02:07.088963 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:07.070888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.2.205.129:37135)
0929 03:02:07.071812 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:07.071818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:07.072217 (+ 399us) server_negotiation.cc:407] Connection header received
0929 03:02:07.072506 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:07.072511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:07.072596 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:07.072691 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:07.074350 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.075524 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.077355 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.078010 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.078114 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:07.083120 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:07.083138 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:07.083157 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:07.083201 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:07.086243 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.087046 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.087055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.087057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.087208 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.087539 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.087542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.087544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.087919 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:07.088145 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:07.088420 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 03:02:07.088608 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751}
I20260929 03:02:07.089627 10676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:07.214272 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10545
I20260929 03:02:07.234162 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10685
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.130@KRBTEST.COM
I20260929 03:02:07.250866 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:07.235325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.2.205.130:41895)
0929 03:02:07.235888 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:07.235895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:07.235916 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:02:07.235981 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:07.235986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:07.236080 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:07.236200 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:07.237639 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.238568 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.240345 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.240809 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.240941 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:07.245188 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:07.245206 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:07.245225 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:07.245270 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:07.248173 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.248924 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.248933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.248935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.249082 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.249430 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.249435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.249437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.249860 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:07.250111 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:07.250337 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:02:07.250537 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":297,"threads_started":1}
I20260929 03:02:07.251600 10815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:07.372826 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10685
I20260929 03:02:07.391688 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10824
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[10429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.205.131@KRBTEST.COM
I20260929 03:02:07.409950 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:07.392617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.2.205.131:43533)
0929 03:02:07.392811 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:07.392818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:07.392839 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:02:07.392957 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:07.392961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:07.393040 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:07.393122 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:07.394314 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.395051 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.396207 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:07.396632 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:07.396723 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:07.404625 (+ 7902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:07.404643 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:07.404660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:07.404708 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:07.407343 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.408003 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.408010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.408012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.408154 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:07.408469 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:07.408472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:07.408474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:07.408942 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:07.409181 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:07.409426 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:02:07.409619 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:02:07.410579 10954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:07.554857 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10824
I20260929 03:02:07.576123 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 10442
I20260929 03:02:07.577353 10514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:07.693361 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 10442
2026-09-29T03:02:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7015 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[11103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[11103](info): set up 2 sockets
Sep 29 03:02:07 dist-test-slave-530n krb5kdc[11103](info): commencing operation
krb5kdc: starting...
W20260929 03:02:08.697816 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:02:08 dist-test-slave-530n krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:02:08.905094 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45991
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:09.226670 11116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:09.226985 11116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:09.227118 11116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:09.236383 11116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:09.236506 11116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:09.236575 11116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:09.236653 11116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:09.236717 11116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:09.247216 11116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:09.248525 11116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:09.249853 11116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:09.261158 11124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:09.261201 11122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:09.261222 11121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:09.262089 11116 server_base.cc:1034] running on GCE node
I20260929 03:02:09.262797 11116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:09.264073 11116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:09.265270 11116 hybrid_clock.cc:648] HybridClock initialized: now 1790650929265224 us; error 26 us; skew 500 ppm
I20260929 03:02:09.265657 11116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:02:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1644 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:02:09 dist-test-slave-530n krb5kdc[11130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:02:09 dist-test-slave-530n krb5kdc[11130](info): set up 2 sockets
Sep 29 03:02:09 dist-test-slave-530n krb5kdc[11130](info): commencing operation
krb5kdc: starting...
W20260929 03:02:10.335053 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:02:10 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:02:10.546481 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35857
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:10.866539 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:10.866801 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:10.866892 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:10.873872 11143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:10.873966 11143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:10.874008 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:10.874061 11143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:10.874102 11143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:10.884716 11143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:10.886112 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:10.887420 11143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:10.899138 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:10.899216 11149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:10.899173 11151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:10.900134 11143 server_base.cc:1034] running on GCE node
I20260929 03:02:10.901022 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:10.902522 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:10.903775 11143 hybrid_clock.cc:648] HybridClock initialized: now 1790650930903714 us; error 52 us; skew 500 ppm
I20260929 03:02:10.904168 11143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:10 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:10.907326 11143 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:02:10.908875 11143 webserver.cc:460] Webserver started at http://127.2.205.190:44465/ using document root <none> and password file <none>
I20260929 03:02:10.909560 11143 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:10.909657 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:10.909917 11143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:10.913014 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "e382dcea676d479a9fa1d306c8afe35a"
format_stamp: "Formatted at 2026-09-29 03:02:10 on dist-test-slave-530n"
server_key: "b2a48c045cf3efd5912793d97bb08351"
server_key_iv: "5ae58beea0d5acbfaa748ff0d2e01f40"
server_key_version: "encryptionkey@0"
I20260929 03:02:10.913758 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "e382dcea676d479a9fa1d306c8afe35a"
format_stamp: "Formatted at 2026-09-29 03:02:10 on dist-test-slave-530n"
server_key: "b2a48c045cf3efd5912793d97bb08351"
server_key_iv: "5ae58beea0d5acbfaa748ff0d2e01f40"
server_key_version: "encryptionkey@0"
I20260929 03:02:10.918761 11143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 03:02:10.922191 11158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:10.923584 11143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 03:02:10.923722 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "e382dcea676d479a9fa1d306c8afe35a"
format_stamp: "Formatted at 2026-09-29 03:02:10 on dist-test-slave-530n"
server_key: "b2a48c045cf3efd5912793d97bb08351"
server_key_iv: "5ae58beea0d5acbfaa748ff0d2e01f40"
server_key_version: "encryptionkey@0"
I20260929 03:02:10.923887 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:10.969199 11143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:10.973462 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:10.973758 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:10.991370 11143 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35857
I20260929 03:02:10.991375 11209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35857 every 8 connection(s)
I20260929 03:02:10.992990 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:10.996860 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11143
I20260929 03:02:10.997056 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:02:10.997134 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:10.997696 2870 external_mini_cluster.cc:1428] Setting key 988ea62e76d9c5ffbb0db9f3519aa97b
I20260929 03:02:11.012162 11210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Bootstrap starting.
Sep 29 03:02:11 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:11.016224 11210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:11.017598 11210 log.cc:824] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:11.021090 11210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: No bootstrap required, opened a new log
I20260929 03:02:11.022312 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:10.999647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.2.205.190:35857)
0929 03:02:11.000422 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:11.000435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:11.000489 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:02:11.002071 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:11.002099 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:11.002570 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:11.002945 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:11.004903 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.006258 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.008197 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.009479 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.009710 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:11.013733 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:11.013746 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:11.013774 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:11.013836 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:11.017728 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:11.018554 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:11.018561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:11.018568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:11.018746 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:11.019147 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:11.019152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:11.019155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:11.019856 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:11.020127 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:11.020506 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 03:02:11.020970 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":348,"threads_started":1}
I20260929 03:02:11.029868 11210 raft_consensus.cc:348] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:11.030247 11210 raft_consensus.cc:374] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:11.030320 11210 raft_consensus.cc:729] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e382dcea676d479a9fa1d306c8afe35a, State: Initialized, Role: FOLLOWER
I20260929 03:02:11.030840 11210 consensus_queue.cc:260] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:11.031060 11210 raft_consensus.cc:388] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:11.031154 11210 raft_consensus.cc:482] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:11.031304 11210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:11.033550 11210 raft_consensus.cc:504] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:11.034175 11210 leader_election.cc:302] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e382dcea676d479a9fa1d306c8afe35a; no voters:
I20260929 03:02:11.034759 11210 leader_election.cc:288] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:02:11.034880 11215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:11.035487 11215 raft_consensus.cc:686] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 LEADER]: Becoming Leader. State: Replica: e382dcea676d479a9fa1d306c8afe35a, State: Running, Role: LEADER
I20260929 03:02:11.036034 11215 consensus_queue.cc:237] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:11.036809 11210 sys_catalog.cc:564] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:11.039182 11217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e382dcea676d479a9fa1d306c8afe35a. Latest consensus state: current_term: 1 leader_uuid: "e382dcea676d479a9fa1d306c8afe35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } } }
I20260929 03:02:11.039140 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e382dcea676d479a9fa1d306c8afe35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } } }
I20260929 03:02:11.039608 11217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:11.039629 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:11.041399 11220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:11.047720 11220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:11.056879 11220 catalog_manager.cc:1347] Generated new cluster ID: 95fbdcd5723147728a8cf96f5ed365a9
I20260929 03:02:11.057009 11220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:11.070559 11220 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:02:11.071749 11220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:11.078857 11220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Generated new TSK 0
I20260929 03:02:11.079612 11220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:02:11.136924 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:11.465488 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:11.465757 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:11.465880 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:11.472843 11236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:11.472950 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:11.473104 11236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:11.482803 11236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:11.484277 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:11.485805 11236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:11.498667 11241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:11.498754 11242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:11.498711 11244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:11.499855 11236 server_base.cc:1034] running on GCE node
I20260929 03:02:11.500491 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:11.501583 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:11.502794 11236 hybrid_clock.cc:648] HybridClock initialized: now 1790650931502737 us; error 50 us; skew 500 ppm
I20260929 03:02:11.503120 11236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:11 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:11.506243 11236 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:02:11.507771 11236 webserver.cc:460] Webserver started at http://127.2.205.129:41489/ using document root <none> and password file <none>
I20260929 03:02:11.508493 11236 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:11.508610 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:11.508961 11236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:11.512025 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "21a73349832446eab8482f647d1a9d05"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "a772a4ce2c959ed66826df3fa64ad2ab"
server_key_iv: "d3701e6fb3504c058c6f35ec018d2235"
server_key_version: "encryptionkey@0"
I20260929 03:02:11.512871 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "21a73349832446eab8482f647d1a9d05"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "a772a4ce2c959ed66826df3fa64ad2ab"
server_key_iv: "d3701e6fb3504c058c6f35ec018d2235"
server_key_version: "encryptionkey@0"
I20260929 03:02:11.517980 11236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:02:11.521405 11251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:11.522753 11236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:02:11.522886 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "21a73349832446eab8482f647d1a9d05"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "a772a4ce2c959ed66826df3fa64ad2ab"
server_key_iv: "d3701e6fb3504c058c6f35ec018d2235"
server_key_version: "encryptionkey@0"
I20260929 03:02:11.523056 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:11.562597 11236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:11.566298 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:11.566550 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:11.567430 11236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:11.568722 11236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:11.568822 11236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:11.568939 11236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:11.568998 11236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:11.597761 11236 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37049
I20260929 03:02:11.597802 11363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37049 every 8 connection(s)
I20260929 03:02:11.598970 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:11.600333 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11236
I20260929 03:02:11.600508 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:02:11.601265 2870 external_mini_cluster.cc:1428] Setting key 8d588ee406bfb4fc420cf5158c60f881
Sep 29 03:02:11 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:11.619030 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:11.601873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:40481 (local address 127.2.205.190:35857)
0929 03:02:11.603221 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:11.603226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:11.603245 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:02:11.605160 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:11.605164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:11.605226 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:11.605311 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:11.607161 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.607842 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.609815 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.610209 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.610295 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:11.613528 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:11.613535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:11.613539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:11.613577 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:11.616307 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:11.616960 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:11.616965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:11.616969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:11.617099 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:11.617430 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:11.617435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:11.617438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:11.617744 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:11.617890 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:11.618470 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 03:02:11.618690 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":1186,"threads_started":1}
I20260929 03:02:11.619670 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:11.601907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:40481)
0929 03:02:11.602790 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:11.602849 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:11.604889 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:11.605563 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:11.605573 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:11.606242 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:11.607007 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:11.607030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.607971 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.607976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:11.609703 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:11.609717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:11.610401 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:11.610408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:11.610666 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:11.611301 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:11.611331 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:11.613326 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:11.616422 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:11.616428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:11.616439 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:11.616855 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:11.617184 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:11.617187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:11.617189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:11.617346 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:11.617845 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:11.617851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:11.618129 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 03:02:11.618332 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:02:11.618671 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":327,"threads_started":1}
I20260929 03:02:11.621007 11364 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35857
I20260929 03:02:11.621271 11364 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:11.621917 11364 heartbeater.cc:511] Master 127.2.205.190:35857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:02:11.623847 11175 ts_manager.cc:194] Registered new tserver with Master: 21a73349832446eab8482f647d1a9d05 (127.2.205.129:37049)
I20260929 03:02:11.625290 11175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:40481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:02:11.648183 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:11.950301 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:11.950572 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:11.950681 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:11.958685 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:11.958799 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:11.958920 11370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:11.968701 11370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:11.969928 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:11.971166 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:11.983424 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:11.983448 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:11.983512 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:11.984194 11370 server_base.cc:1034] running on GCE node
I20260929 03:02:11.984825 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:11.985745 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:11.986939 11370 hybrid_clock.cc:648] HybridClock initialized: now 1790650931986892 us; error 47 us; skew 500 ppm
I20260929 03:02:11.987229 11370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:11 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:11.990164 11370 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:02:11.991755 11370 webserver.cc:460] Webserver started at http://127.2.205.130:41271/ using document root <none> and password file <none>
I20260929 03:02:11.992599 11370 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:11.992726 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:11.993093 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:11.996816 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "2bebf01ddb0945a7bbf8842ad97337c5"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "fc3191dd8677fdca64280d762aba2bcf"
server_key_iv: "4e6871ca1de497f6207ff1539b45bcae"
server_key_version: "encryptionkey@0"
I20260929 03:02:11.997763 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "2bebf01ddb0945a7bbf8842ad97337c5"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "fc3191dd8677fdca64280d762aba2bcf"
server_key_iv: "4e6871ca1de497f6207ff1539b45bcae"
server_key_version: "encryptionkey@0"
I20260929 03:02:12.003849 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 03:02:12.007980 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.009469 11370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:12.009611 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "2bebf01ddb0945a7bbf8842ad97337c5"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "fc3191dd8677fdca64280d762aba2bcf"
server_key_iv: "4e6871ca1de497f6207ff1539b45bcae"
server_key_version: "encryptionkey@0"
I20260929 03:02:12.009826 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:12.023206 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:12.027093 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:12.027325 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:12.028194 11370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:12.029486 11370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:12.029557 11370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.029641 11370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:12.029690 11370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.058029 11370 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:38077
I20260929 03:02:12.058076 11497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:38077 every 8 connection(s)
I20260929 03:02:12.059211 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:12.065713 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11370
I20260929 03:02:12.065860 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:02:12.066270 2870 external_mini_cluster.cc:1428] Setting key d61bbbf7ac5dd7e04e02275c009001e5
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:12.077911 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.061490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:44861 (local address 127.2.205.190:35857)
0929 03:02:12.061650 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:12.061654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:12.062735 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 03:02:12.064471 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:12.064477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:12.064555 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:12.064639 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:12.066146 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.066982 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.068860 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.069275 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.069382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.072647 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:12.072653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:12.072656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:12.072687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:12.075109 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.075762 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.075765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.075768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.075889 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.076272 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.076277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.076280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.076558 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:12.076681 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:12.077360 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 03:02:12.077585 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:02:12.078837 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.061919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:44861)
0929 03:02:12.062583 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:12.062622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:12.064216 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:12.064896 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:12.064904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:12.065395 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:12.065992 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:12.066012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.067111 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.067116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:12.068696 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:12.068709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.069477 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.069485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:12.069706 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.070295 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:12.070316 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:12.072444 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:12.075241 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:12.075250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:12.075266 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:12.075654 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:12.075994 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:12.076000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:12.076003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:12.076172 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:12.076675 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:12.076683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:12.077035 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 03:02:12.077270 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 03:02:12.077752 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":244,"threads_started":1}
I20260929 03:02:12.080346 11498 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35857
I20260929 03:02:12.080679 11498 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:12.081580 11498 heartbeater.cc:511] Master 127.2.205.190:35857 requested a full tablet report, sending...
I20260929 03:02:12.083166 11175 ts_manager.cc:194] Registered new tserver with Master: 2bebf01ddb0945a7bbf8842ad97337c5 (127.2.205.130:38077)
I20260929 03:02:12.084246 11175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:44861
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:02:12.107692 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:12.412281 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:12.412559 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:12.412701 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:12.419723 11503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:12.419837 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:12.419996 11503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:12.429600 11503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:12.430812 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:12.432097 11503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:12.444425 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:12.444381 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:12.444350 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:12.444962 11503 server_base.cc:1034] running on GCE node
I20260929 03:02:12.445523 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:12.446452 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:12.447654 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790650932447609 us; error 34 us; skew 500 ppm
I20260929 03:02:12.447958 11503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:12.454651 11503 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:02:12.456060 11503 webserver.cc:460] Webserver started at http://127.2.205.131:38281/ using document root <none> and password file <none>
I20260929 03:02:12.456730 11503 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:12.456895 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:12.457190 11503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:12.459969 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "1f4204e6a277441daab0450dde513aec"
format_stamp: "Formatted at 2026-09-29 03:02:12 on dist-test-slave-530n"
server_key: "9d44ddfc22aab5265308691100a3a8d5"
server_key_iv: "f6df9cb55870cbc5e6628f4be9b0cdf9"
server_key_version: "encryptionkey@0"
I20260929 03:02:12.460686 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "1f4204e6a277441daab0450dde513aec"
format_stamp: "Formatted at 2026-09-29 03:02:12 on dist-test-slave-530n"
server_key: "9d44ddfc22aab5265308691100a3a8d5"
server_key_iv: "f6df9cb55870cbc5e6628f4be9b0cdf9"
server_key_version: "encryptionkey@0"
I20260929 03:02:12.465582 11503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 03:02:12.468732 11518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.470029 11503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:02:12.470180 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "1f4204e6a277441daab0450dde513aec"
format_stamp: "Formatted at 2026-09-29 03:02:12 on dist-test-slave-530n"
server_key: "9d44ddfc22aab5265308691100a3a8d5"
server_key_iv: "f6df9cb55870cbc5e6628f4be9b0cdf9"
server_key_version: "encryptionkey@0"
I20260929 03:02:12.470350 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:12.494866 11503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:12.498539 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:12.498780 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:12.499648 11503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:12.500880 11503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:12.500953 11503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.501066 11503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:12.501120 11503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:12.529839 11503 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:42417
I20260929 03:02:12.529882 11630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:42417 every 8 connection(s)
I20260929 03:02:12.531056 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:12.535523 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11503
I20260929 03:02:12.535688 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:02:12.536108 2870 external_mini_cluster.cc:1428] Setting key b76ef7d608809f0c7922433b2a8982ff
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:12.549929 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.533463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:43321 (local address 127.2.205.190:35857)
0929 03:02:12.533619 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:12.533623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:12.534778 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 03:02:12.536587 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:12.536593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:12.536705 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:12.536833 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:12.538671 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.539467 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.541127 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.541531 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.541617 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.544652 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:12.544657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:12.544660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:12.544692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:12.547173 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.547830 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.547833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.547836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.547943 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.548295 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.548299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.548302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.548606 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:12.548722 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:12.549394 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 03:02:12.549577 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:02:12.551031 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.533956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:43321)
0929 03:02:12.534631 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:12.534673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:12.536328 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:12.537082 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:12.537094 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:12.537735 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:12.538510 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:12.538536 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.539605 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.539609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:12.541014 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:12.541026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.541696 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.541701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:12.541887 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.542469 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:12.542491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:12.544463 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:12.547288 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:12.547294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:12.547306 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:12.547736 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:12.548014 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:12.548017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:12.548019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:12.548183 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:12.548709 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:12.548714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:12.549042 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 03:02:12.549285 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 03:02:12.549725 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260929 03:02:12.552263 11631 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35857
I20260929 03:02:12.552538 11631 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:12.553228 11631 heartbeater.cc:511] Master 127.2.205.190:35857 requested a full tablet report, sending...
I20260929 03:02:12.554641 11175 ts_manager.cc:194] Registered new tserver with Master: 1f4204e6a277441daab0450dde513aec (127.2.205.131:42417)
I20260929 03:02:12.555398 11175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:43321
I20260929 03:02:12.563261 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
Entry for principal oryx/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab.
WARNING: no policy specified for oryx/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.130@KRBTEST.COM" created.
I20260929 03:02:12.628234 11364 heartbeater.cc:503] Master 127.2.205.190:35857 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
Entry for principal oryx/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab.
WARNING: no policy specified for oryx/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
Entry for principal oryx/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab.
I20260929 03:02:12.680421 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11236
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:02:12.695333 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.681463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.2.205.129:37049)
0929 03:02:12.682132 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:12.682140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:12.682166 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:02:12.682244 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:12.682251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:12.682481 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:12.682623 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:12.683879 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.684909 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.686178 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.686715 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.686803 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.690520 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:12.690532 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:12.690546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:12.690585 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:12.693217 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.693737 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.693743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.693745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.693858 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.694134 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.694136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.694138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.694442 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:12.694658 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:12.694867 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 03:02:12.695063 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":317,"threads_started":1}
I20260929 03:02:12.696542 11359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:12.776861 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11236
I20260929 03:02:12.789949 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11370
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:02:12.804454 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.790754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.2.205.130:38077)
0929 03:02:12.791235 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:12.791241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:12.791264 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:02:12.791320 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:12.791327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:12.791535 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:12.791673 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:12.792843 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.793810 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.795008 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.795549 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.795657 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.799400 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:12.799410 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:12.799422 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:12.799453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:12.801859 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.802466 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.802479 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.802481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.802598 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.802927 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.802931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.802933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.803240 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:12.803533 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:12.803794 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 03:02:12.804034 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260929 03:02:12.805845 11493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:12.886449 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11370
I20260929 03:02:12.899411 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11503
Sep 29 03:02:12 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:02:12.914021 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:12.900248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.2.205.131:42417)
0929 03:02:12.900829 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:12.900836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:12.900864 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:02:12.900926 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:12.900932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:12.901168 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:12.901289 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:12.902446 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.903424 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.904597 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:12.905214 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:12.905303 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:12.909183 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:12.909193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:12.909207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:12.909241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:12.911770 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.912321 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.912327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.912330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.912457 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:12.912705 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:12.912708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:12.912710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:12.913071 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:12.913289 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:12.913493 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:02:12.913682 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":255,"threads_started":1}
I20260929 03:02:12.915169 11626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:12.995208 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11503
I20260929 03:02:13.007529 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11143
I20260929 03:02:13.008679 11205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:13.108886 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11143
I20260929 03:02:13.125655 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35857
--webserver_interface=127.2.205.190
--webserver_port=44465
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:13.434137 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:13.434403 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:13.434558 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:13.441766 11648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:13.441874 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:13.441927 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:13.442010 11648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:13.442063 11648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:13.451733 11648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=44465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:13.453034 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:13.454255 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:13.465107 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:13.465174 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:13.465152 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:13.466224 11648 server_base.cc:1034] running on GCE node
I20260929 03:02:13.466876 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:13.468138 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:13.469331 11648 hybrid_clock.cc:648] HybridClock initialized: now 1790650933469277 us; error 41 us; skew 500 ppm
I20260929 03:02:13.469640 11648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:13 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650933, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:13.472489 11648 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:02:13.473817 11648 webserver.cc:460] Webserver started at http://127.2.205.190:44465/ using document root <none> and password file <none>
I20260929 03:02:13.474449 11648 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:13.474568 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:13.479784 11648 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 03:02:13.482409 11663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:13.483680 11648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:02:13.483819 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "e382dcea676d479a9fa1d306c8afe35a"
format_stamp: "Formatted at 2026-09-29 03:02:10 on dist-test-slave-530n"
server_key: "b2a48c045cf3efd5912793d97bb08351"
server_key_iv: "5ae58beea0d5acbfaa748ff0d2e01f40"
server_key_version: "encryptionkey@0"
I20260929 03:02:13.484694 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:13.500660 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:13.504511 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:13.504711 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:13.520313 11648 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35857
I20260929 03:02:13.520332 11714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35857 every 8 connection(s)
I20260929 03:02:13.521687 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:13.523989 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11648
I20260929 03:02:13.524880 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:37049
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:35857
--webserver_port=41489
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:02:13.528548 11715 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:02:13.532313 11715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Bootstrap starting.
I20260929 03:02:13.557993 11715 log.cc:824] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:13.568893 11715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:02:13.569530 11715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Bootstrap complete.
I20260929 03:02:13.582782 11715 raft_consensus.cc:348] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:13.584012 11715 raft_consensus.cc:729] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e382dcea676d479a9fa1d306c8afe35a, State: Initialized, Role: FOLLOWER
I20260929 03:02:13.584780 11715 consensus_queue.cc:260] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:13.585086 11715 raft_consensus.cc:388] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:13.585206 11715 raft_consensus.cc:482] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:13.585376 11715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:02:13.591356 11715 raft_consensus.cc:504] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:13.591933 11715 leader_election.cc:302] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e382dcea676d479a9fa1d306c8afe35a; no voters:
I20260929 03:02:13.592597 11715 leader_election.cc:288] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:02:13.592816 11719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:02:13.593871 11719 raft_consensus.cc:686] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [term 2 LEADER]: Becoming Leader. State: Replica: e382dcea676d479a9fa1d306c8afe35a, State: Running, Role: LEADER
I20260929 03:02:13.594416 11719 consensus_queue.cc:237] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } }
I20260929 03:02:13.594666 11715 sys_catalog.cc:564] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:13.596585 11721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e382dcea676d479a9fa1d306c8afe35a. Latest consensus state: current_term: 2 leader_uuid: "e382dcea676d479a9fa1d306c8afe35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } } }
I20260929 03:02:13.596930 11721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:13.597427 11720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e382dcea676d479a9fa1d306c8afe35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e382dcea676d479a9fa1d306c8afe35a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35857 } } }
I20260929 03:02:13.597718 11720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:13.599704 11725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:13.608842 11725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:13.610374 11725 catalog_manager.cc:1259] Loaded cluster ID: 95fbdcd5723147728a8cf96f5ed365a9
I20260929 03:02:13.610538 11725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:13.614686 11725 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:13.616279 11725 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e382dcea676d479a9fa1d306c8afe35a: Loaded TSK: 0
I20260929 03:02:13.617906 11725 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:02:13.872015 11717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:13.872272 11717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:13.872385 11717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:13.879572 11717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:13.879674 11717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:13.879810 11717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:13.890236 11717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.129.keytab
--principal=oryx/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:37049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=41489
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:13.892019 11717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:13.893365 11717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:13.908079 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:13.908058 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:13.909276 11717 server_base.cc:1034] running on GCE node
W20260929 03:02:13.909354 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:13.910086 11717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:13.911098 11717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:13.912308 11717 hybrid_clock.cc:648] HybridClock initialized: now 1790650933912259 us; error 49 us; skew 500 ppm
I20260929 03:02:13.912607 11717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:13 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:13.915526 11717 init.cc:377] Logged in from keytab as oryx/127.2.205.129@KRBTEST.COM (short username oryx)
I20260929 03:02:13.917003 11717 webserver.cc:460] Webserver started at http://127.2.205.129:41489/ using document root <none> and password file <none>
I20260929 03:02:13.917636 11717 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:13.917754 11717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:13.923089 11717 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:02:13.926342 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:13.927912 11717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:02:13.928081 11717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "21a73349832446eab8482f647d1a9d05"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "a772a4ce2c959ed66826df3fa64ad2ab"
server_key_iv: "d3701e6fb3504c058c6f35ec018d2235"
server_key_version: "encryptionkey@0"
I20260929 03:02:13.929355 11717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:13.950371 11717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:13.955305 11717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:13.955591 11717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:13.956696 11717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:13.958280 11717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:13.958392 11717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:13.958487 11717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:13.958557 11717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:13.986821 11717 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37049
I20260929 03:02:13.986873 11863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37049 every 8 connection(s)
I20260929 03:02:13.988219 11717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:13.998034 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11717
I20260929 03:02:13.998608 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:38077
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:35857
--webserver_port=41271
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:02:13 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:13 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:14.310680 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:14.310958 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:14.311096 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:14.318186 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:14.318298 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:14.318440 11868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:14.327919 11868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.130.keytab
--principal=oryx/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:38077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=41271
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:14.329197 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:14.330406 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:14.345047 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:14.344947 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:14.345108 11877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:14.345768 11868 server_base.cc:1034] running on GCE node
I20260929 03:02:14.346316 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:14.347213 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:14.348407 11868 hybrid_clock.cc:648] HybridClock initialized: now 1790650934348345 us; error 63 us; skew 500 ppm
I20260929 03:02:14.348699 11868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650934, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:14.351899 11868 init.cc:377] Logged in from keytab as oryx/127.2.205.130@KRBTEST.COM (short username oryx)
I20260929 03:02:14.353400 11868 webserver.cc:460] Webserver started at http://127.2.205.130:41271/ using document root <none> and password file <none>
I20260929 03:02:14.354053 11868 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:14.354177 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:14.359220 11868 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:02:14.361714 11884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.362990 11868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:14.363126 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "2bebf01ddb0945a7bbf8842ad97337c5"
format_stamp: "Formatted at 2026-09-29 03:02:11 on dist-test-slave-530n"
server_key: "fc3191dd8677fdca64280d762aba2bcf"
server_key_iv: "4e6871ca1de497f6207ff1539b45bcae"
server_key_version: "encryptionkey@0"
I20260929 03:02:14.364003 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:14.376821 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:14.380440 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:14.380661 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:14.381590 11868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:14.382858 11868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:14.382949 11868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.383046 11868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:14.383112 11868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.418886 11868 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:38077
I20260929 03:02:14.418922 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:38077 every 8 connection(s)
I20260929 03:02:14.420298 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:14.430637 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 11868
I20260929 03:02:14.431432 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:42417
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:35857
--webserver_port=38281
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:14.753486 12001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:14.753756 12001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:14.753899 12001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:14.761025 12001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:14.761147 12001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:14.761299 12001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:14.770905 12001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/krb5kdc/oryx_127.2.205.131.keytab
--principal=oryx/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:42417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=38281
--tserver_master_addrs=127.2.205.190:35857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:14.772156 12001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:14.773593 12001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:14.787912 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:14.788295 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:14.798033 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:14.798581 12001 server_base.cc:1034] running on GCE node
I20260929 03:02:14.799304 12001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:14.800254 12001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:14.801517 12001 hybrid_clock.cc:648] HybridClock initialized: now 1790650934801456 us; error 53 us; skew 500 ppm
I20260929 03:02:14.801815 12001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650934, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:14.804740 12001 init.cc:377] Logged in from keytab as oryx/127.2.205.131@KRBTEST.COM (short username oryx)
I20260929 03:02:14.806357 12001 webserver.cc:460] Webserver started at http://127.2.205.131:38281/ using document root <none> and password file <none>
I20260929 03:02:14.807029 12001 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:14.807163 12001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:14.812274 12001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:14.814829 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.816114 12001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:14.816257 12001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "1f4204e6a277441daab0450dde513aec"
format_stamp: "Formatted at 2026-09-29 03:02:12 on dist-test-slave-530n"
server_key: "9d44ddfc22aab5265308691100a3a8d5"
server_key_iv: "f6df9cb55870cbc5e6628f4be9b0cdf9"
server_key_version: "encryptionkey@0"
I20260929 03:02:14.817188 12001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:14.837174 12001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:14.841126 12001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:14.841359 12001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:14.842319 12001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:14.843683 12001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:14.843773 12001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.843880 12001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:14.843946 12001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:14.891424 12001 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:42417
I20260929 03:02:14.891466 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:42417 every 8 connection(s)
I20260929 03:02:14.893288 12001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:14.897061 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12001
I20260929 03:02:14.933593 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:14.898137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.2.205.190:35857)
0929 03:02:14.905163 (+ 7026us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:14.905169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:14.912794 (+ 7625us) server_negotiation.cc:407] Connection header received
0929 03:02:14.912894 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:14.912901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:14.913003 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:14.913137 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:14.915116 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.916454 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.918617 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.919353 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.919490 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:14.924333 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:14.924345 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:14.924368 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:14.924422 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:14.928644 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:14.929587 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:14.929598 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:14.929609 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:14.929890 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:14.930321 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:14.930326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:14.930330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:14.931021 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:14.931364 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:14.931818 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 03:02:14.932362 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6889,"thread_start_us":6760,"threads_started":1}
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:14 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:15.179383 11867 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.178s user 0.004s sys 0.010s
W20260929 03:02:15.181742 11866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:13.991178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:39113 (local address 127.2.205.190:35857)
0929 03:02:13.991900 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:13.991917 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:13.992159 (+ 242us) server_negotiation.cc:407] Connection header received
0929 03:02:13.994254 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:13.994268 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:13.994591 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:13.994900 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:13.996338 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:13.997616 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:13.999158 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:13.999782 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:13.999970 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.181334 (+1181364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:39113: BlockingRecv error: recv got EOF from 127.2.205.129:39113 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":250,"threads_started":1}
I20260929 03:02:15.181843 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:13.991263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:39113)
0929 03:02:13.992008 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:13.992044 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:13.993728 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:13.995060 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:13.995070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:13.995649 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:13.996164 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:13.996181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:13.997761 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:13.997765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:13.999031 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:13.999043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:13.999998 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.000006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:14.000255 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:14.000885 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:14.000906 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.180161 (+1179255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":320,"threads_started":1}
W20260929 03:02:15.183825 11864 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.205.190:35857: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:15.201525 11866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:15.185109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:43325 (local address 127.2.205.190:35857)
0929 03:02:15.186672 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:15.186678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:15.186696 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:02:15.186749 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:15.186753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:15.186826 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:15.186916 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:15.190464 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.191709 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.194716 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.195331 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.195547 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.201272 (+ 5725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:43325: BlockingRecv error: recv got EOF from 127.2.205.129:43325 (error 108)
Metrics: {"server-negotiator.queue_time_us":415}
I20260929 03:02:15.202629 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:15.185351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:43325)
0929 03:02:15.185747 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:15.185768 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:15.185883 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:15.187120 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:15.187125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:15.187611 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:15.190311 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.190326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.193553 (+ 3227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.193560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.194593 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.194609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.195855 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.195862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.196071 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.196767 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:15.196791 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.202296 (+ 5505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232}
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:02:15.214977 11866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:15.203781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:36735 (local address 127.2.205.190:35857)
0929 03:02:15.203924 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:15.203929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:15.204089 (+ 160us) server_negotiation.cc:407] Connection header received
0929 03:02:15.204190 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:15.204195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:15.204264 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:15.204350 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:15.205332 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.206893 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.208795 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.209399 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.209486 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.214718 (+ 5232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:36735: BlockingRecv error: recv got EOF from 127.2.205.129:36735 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:02:15.215657 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:15.203449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:36735)
0929 03:02:15.203953 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:15.203969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:15.204085 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:15.204372 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:15.204376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:15.204911 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:15.205174 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.205187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.207109 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.207114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.208618 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.208634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.209781 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.209787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.209965 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.210601 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:15.210624 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.215369 (+ 4745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":451,"threads_started":1}
W20260929 03:02:15.216128 11864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:02:15.680343 12000 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.248s user 0.004s sys 0.007s
W20260929 03:02:15.681782 11999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:14.423118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:54953 (local address 127.2.205.190:35857)
0929 03:02:14.423548 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:14.423552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:14.424813 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 03:02:14.426669 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:14.426675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:14.426756 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:14.426857 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:14.428302 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.429091 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.430574 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.431033 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.431145 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.681553 (+1250408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:54953: BlockingRecv error: recv got EOF from 127.2.205.130:54953 (error 108)
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":226,"threads_started":1}
I20260929 03:02:15.682803 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:14.423869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:54953)
0929 03:02:14.424642 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:14.424687 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:14.426425 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:14.427089 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:14.427098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:14.427600 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:14.428142 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:14.428158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.429231 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.429235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:14.430437 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:14.430449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.431242 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.431250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:14.431483 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:14.432156 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:14.432178 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.681438 (+1249260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":309,"threads_started":1}
W20260929 03:02:15.683714 11997 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.205.190:35857: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:15.694517 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:15.684347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:36339)
0929 03:02:15.684570 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:15.684585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:15.684673 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:15.685340 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:15.685347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:15.685861 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:15.686104 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.686116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.687131 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.687137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.688918 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.688935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.689656 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.689662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.689851 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.690468 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:15.690492 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.694255 (+ 3763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131}
W20260929 03:02:15.694670 11999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:15.684902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:36339 (local address 127.2.205.190:35857)
0929 03:02:15.685053 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:15.685057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:15.685071 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:02:15.685114 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:15.685118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:15.685179 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:15.685271 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:15.686247 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.687003 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.689063 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.689475 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.689590 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.694442 (+ 4852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:36339: BlockingRecv error: recv got EOF from 127.2.205.130:36339 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:15 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:15.705175 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:15.695190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:54413)
0929 03:02:15.695320 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:15.695337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:15.695424 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:15.696065 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:15.696071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:15.696505 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:15.696711 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.696722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.697690 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.697696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.699309 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:15.699323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.699959 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.699965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:15.700158 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.700721 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:15.700773 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:15.704868 (+ 4095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:02:15.705281 11999 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:15.695405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:54413 (local address 127.2.205.190:35857)
0929 03:02:15.695573 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:15.695579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:15.695597 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:02:15.695651 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:15.695657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:15.695775 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:15.695890 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:15.696856 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.697571 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.699455 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:15.699833 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:15.699915 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:15.705043 (+ 5128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:54413: BlockingRecv error: recv got EOF from 127.2.205.130:54413 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 03:02:15.705646 11997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:02:16.084218 12135 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.148s user 0.004s sys 0.014s
W20260929 03:02:16.085335 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:14.907758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:34509 (local address 127.2.205.190:35857)
0929 03:02:14.914788 (+ 7030us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:14.914794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:14.914816 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:02:14.919775 (+ 4959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:14.919780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:14.919856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:14.919949 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:14.922705 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.924034 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.933852 (+ 9818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.934624 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.934729 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.085130 (+1150401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:34509: BlockingRecv error: recv got EOF from 127.2.205.131:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":6892,"thread_start_us":6772,"threads_started":1}
I20260929 03:02:16.085815 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:14.908610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:34509)
0929 03:02:14.913959 (+ 5349us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:14.914022 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:14.918283 (+ 4261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:14.920635 (+ 2352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:14.920650 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:14.921530 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:14.922482 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:14.922510 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.925414 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.925420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:14.933652 (+ 8232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:14.933673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:14.935181 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:14.935191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:14.935535 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:14.936397 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:14.936440 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:16.084944 (+1148504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4996,"thread_start_us":4732,"threads_started":1}
W20260929 03:02:16.086520 12130 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35857 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.205.190:35857: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:16.096110 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:16.087101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:56863)
0929 03:02:16.087284 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:16.087301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:16.087395 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:16.087691 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:16.087694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:16.088030 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:16.088206 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.088214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.089336 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.089340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.090389 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.090411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.091167 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.091172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.091301 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.091976 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:16.091995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:16.095892 (+ 3897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:02:16.096261 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:16.087192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:56863 (local address 127.2.205.190:35857)
0929 03:02:16.087354 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:16.087358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:16.087375 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:02:16.087531 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:16.087535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:16.087599 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:16.087676 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:16.088334 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.089183 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.090564 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.090990 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.091085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.096073 (+ 4988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:56863: BlockingRecv error: recv got EOF from 127.2.205.131:56863 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:16.104844 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:16.096779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:59029)
0929 03:02:16.096930 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:16.096950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:16.097055 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:16.097295 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:16.097298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:16.097592 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:16.097742 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.097750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.098764 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.098768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.099728 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.099740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.100485 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.100490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.100632 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.101241 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:16.101260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:16.104602 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 03:02:16.105122 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:16.096846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:59029 (local address 127.2.205.190:35857)
0929 03:02:16.097070 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:16.097075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:16.097107 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:02:16.097148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:16.097151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:16.097217 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:16.097281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:16.097860 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.098626 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.099896 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.100300 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.100410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.104876 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:59029: BlockingRecv error: recv got EOF from 127.2.205.131:59029 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260929 03:02:16.105170 12130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:16.225689 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:16.216731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:47845)
0929 03:02:16.217251 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:16.217270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:16.217402 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:16.217717 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:16.217722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:16.218132 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:16.218291 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.218301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.219334 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.219339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.220350 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.220364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.221130 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.221136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.221307 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.221917 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:16.221936 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:16.225392 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":255,"threads_started":1}
W20260929 03:02:16.225826 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:16.217005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:47845 (local address 127.2.205.190:35857)
0929 03:02:16.217218 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:16.217224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:16.217342 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:02:16.217522 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:16.217528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:16.217592 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:16.217671 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:16.218429 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.219185 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.220486 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.220969 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.221100 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.225617 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:47845: BlockingRecv error: recv got EOF from 127.2.205.129:47845 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:16 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:16.715274 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:16.706296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:46911)
0929 03:02:16.706894 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:16.706910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:16.707050 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:16.707357 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:16.707361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:16.707798 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:16.707946 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.707955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.709029 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.709033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.709981 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:16.709995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.710739 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.710745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:16.710896 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.711541 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:16.711561 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:16.715035 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":373,"threads_started":1}
W20260929 03:02:16.715474 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:16.706446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:46911 (local address 127.2.205.190:35857)
0929 03:02:16.706653 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:16.706660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:16.707005 (+ 345us) server_negotiation.cc:407] Connection header received
0929 03:02:16.707164 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:16.707169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:16.707229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:16.707306 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:16.708094 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.708896 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.710138 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:16.710565 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:16.710673 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:16.715247 (+ 4574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:46911: BlockingRecv error: recv got EOF from 127.2.205.130:46911 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:17.114619 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:17.105763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:51105)
0929 03:02:17.106292 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:17.106308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:17.106436 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:17.106746 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:17.106750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:17.107184 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:17.107334 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.107342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.108397 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.108401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.109415 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.109430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.110159 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.110164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.110281 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.110843 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:17.110861 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:17.114329 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":299,"threads_started":1}
W20260929 03:02:17.114827 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:17.105934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:51105 (local address 127.2.205.190:35857)
0929 03:02:17.106096 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:17.106100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:17.106400 (+ 300us) server_negotiation.cc:407] Connection header received
0929 03:02:17.106537 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:17.106541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:17.106624 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:17.106721 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:17.107506 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.108252 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.109554 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.109999 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.110093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.114575 (+ 4482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:51105: BlockingRecv error: recv got EOF from 127.2.205.131:51105 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:17.236524 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:17.226746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:43311)
0929 03:02:17.227179 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:17.227195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:17.227336 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:17.227687 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:17.227692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:17.228140 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:17.228341 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.228355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.229352 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.229355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.230328 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.230339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.231273 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.231279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.231451 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.232163 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:17.232187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:17.236164 (+ 3977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
W20260929 03:02:17.236863 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:17.226808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:43311 (local address 127.2.205.190:35857)
0929 03:02:17.227033 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:17.227039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:17.227259 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:02:17.227454 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:17.227458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:17.227524 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:17.227623 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:17.228489 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.229253 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.230481 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.231135 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.231264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.236574 (+ 5310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:43311: BlockingRecv error: recv got EOF from 127.2.205.129:43311 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:17 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:17.729447 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:17.716168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:51449)
0929 03:02:17.716649 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:17.716664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:17.716829 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:17.717150 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:17.717155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:17.717585 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:17.717762 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.717771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.718719 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.718723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.719639 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:17.719651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.720318 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.720324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:17.720505 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.725495 (+ 4990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:17.725516 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:17.729183 (+ 3667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
W20260929 03:02:17.729588 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:17.716249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:51449 (local address 127.2.205.190:35857)
0929 03:02:17.716477 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:17.716483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:17.716770 (+ 287us) server_negotiation.cc:407] Connection header received
0929 03:02:17.716940 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:17.716945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:17.717028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:17.717105 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:17.717892 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.718590 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.719767 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:17.720180 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:17.720283 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:17.729373 (+ 9090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:51449: BlockingRecv error: recv got EOF from 127.2.205.130:51449 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:18.125047 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:18.115739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:54553)
0929 03:02:18.116224 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:18.116242 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:18.116369 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:18.116728 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:18.116732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:18.117153 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:18.117352 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.117362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.118508 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.118514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.119466 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.119479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.120131 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.120135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.120301 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.120933 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:18.120952 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:18.124644 (+ 3692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
W20260929 03:02:18.125146 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:18.115905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:54553 (local address 127.2.205.190:35857)
0929 03:02:18.116070 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:18.116075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:18.116308 (+ 233us) server_negotiation.cc:407] Connection header received
0929 03:02:18.116461 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:18.116464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:18.116549 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:18.116630 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:18.117543 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.118340 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.119606 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.120009 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.120102 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.124906 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:54553: BlockingRecv error: recv got EOF from 127.2.205.131:54553 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:18.246604 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:18.237824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:45651)
0929 03:02:18.238320 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:18.238337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:18.238456 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:18.238795 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:18.238800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:18.239206 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:18.239395 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.239405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.240460 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.240465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.241471 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.241485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.242228 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.242232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.242380 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.242961 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:18.242979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:18.246264 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
W20260929 03:02:18.246731 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:18.237859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45651 (local address 127.2.205.190:35857)
0929 03:02:18.238051 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:18.238055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:18.238416 (+ 361us) server_negotiation.cc:407] Connection header received
0929 03:02:18.238550 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:18.238554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:18.238629 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:18.238709 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:18.239542 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.240304 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.241633 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.242090 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.242198 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.246505 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:45651: BlockingRecv error: recv got EOF from 127.2.205.129:45651 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:18 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:18.739185 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:18.730342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:51011)
0929 03:02:18.730806 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:18.730822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:18.730935 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:18.731229 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:18.731232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:18.731572 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:18.731744 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.731752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.732712 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.732716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.733656 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:18.733669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.734363 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.734367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:18.734522 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.735183 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:18.735202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:18.738923 (+ 3721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
W20260929 03:02:18.739378 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:18.730443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:51011 (local address 127.2.205.190:35857)
0929 03:02:18.730661 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:18.730667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:18.730893 (+ 226us) server_negotiation.cc:407] Connection header received
0929 03:02:18.731036 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:18.731040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:18.731103 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:18.731180 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:18.731858 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.732583 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.733815 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:18.734246 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:18.734339 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:18.739144 (+ 4805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:51011: BlockingRecv error: recv got EOF from 127.2.205.130:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:19.135802 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:19.126294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:54251)
0929 03:02:19.126951 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:19.127023 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:19.127195 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:19.127552 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:19.127557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:19.127913 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:19.128093 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.128102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.129348 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.129354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.130608 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.130625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.131351 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.131357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.131533 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.132096 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:19.132115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:19.135557 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":406,"threads_started":1}
W20260929 03:02:19.135934 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:19.126466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:54251 (local address 127.2.205.190:35857)
0929 03:02:19.126772 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:19.126777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:19.127120 (+ 343us) server_negotiation.cc:407] Connection header received
0929 03:02:19.127302 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:19.127306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:19.127375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:19.127467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:19.128221 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.129183 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.130730 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.131215 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.131325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.135756 (+ 4431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:54251: BlockingRecv error: recv got EOF from 127.2.205.131:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:19.256186 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:19.247699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:45043)
0929 03:02:19.248218 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:19.248234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:19.248345 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:19.248628 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:19.248631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:19.249056 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:19.249213 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.249223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.250196 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.250199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.251194 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.251205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.251913 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.251919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.252091 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.252689 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:19.252707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:19.255943 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
W20260929 03:02:19.256357 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:19.247826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45043 (local address 127.2.205.190:35857)
0929 03:02:19.248013 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:19.248017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:19.248302 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:02:19.248447 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:19.248451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:19.248513 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:19.248583 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:19.249330 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.250091 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.251340 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.251762 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.251892 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.256156 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:45043: BlockingRecv error: recv got EOF from 127.2.205.129:45043 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:19 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:19.748967 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:19.740163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:60681)
0929 03:02:19.740773 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:19.740790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:19.740902 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:19.741177 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:19.741180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:19.741645 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:19.741817 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.741827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.742828 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.742832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.743835 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:19.743847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.744513 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.744517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:19.744639 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.745235 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:19.745253 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:19.748598 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
W20260929 03:02:19.749043 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:19.740238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:60681 (local address 127.2.205.190:35857)
0929 03:02:19.740392 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:19.740396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:19.740853 (+ 457us) server_negotiation.cc:407] Connection header received
0929 03:02:19.740987 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:19.740990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:19.741060 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:19.741139 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:19.741957 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.742729 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.743951 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:19.744384 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:19.744475 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:19.748823 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:60681: BlockingRecv error: recv got EOF from 127.2.205.130:60681 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:20.145561 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:20.136848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:56897)
0929 03:02:20.137383 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:20.137398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:20.137512 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:20.137793 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:20.137796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:20.138191 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:20.138358 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.138367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.139395 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.139401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.140434 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.140447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.141128 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.141133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.141268 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.141858 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:20.141877 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:20.145301 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":308,"threads_started":1}
W20260929 03:02:20.145747 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:20.136912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:56897 (local address 127.2.205.190:35857)
0929 03:02:20.137118 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:20.137125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:20.137474 (+ 349us) server_negotiation.cc:407] Connection header received
0929 03:02:20.137604 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:20.137607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:20.137685 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:20.137761 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:20.138504 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.139237 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.140555 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.140967 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.141076 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.145529 (+ 4453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:56897: BlockingRecv error: recv got EOF from 127.2.205.131:56897 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:20.265213 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:20.257106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:50603)
0929 03:02:20.257615 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:20.257631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:20.257760 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:20.258089 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:20.258093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:20.258508 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:20.258669 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.258678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.259695 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.259700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.260641 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.260654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.261310 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.261314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.261448 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.261950 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:20.261967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:20.264980 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
W20260929 03:02:20.265367 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:20.257165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:50603 (local address 127.2.205.190:35857)
0929 03:02:20.257356 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:20.257363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:20.257709 (+ 346us) server_negotiation.cc:407] Connection header received
0929 03:02:20.257861 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:20.257867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:20.257946 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:20.258023 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:20.258828 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.259566 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.260793 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.261188 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.261279 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.265165 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:50603: BlockingRecv error: recv got EOF from 127.2.205.129:50603 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:20 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:20.758946 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:20.749972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:60293)
0929 03:02:20.750516 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:20.750535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:20.750670 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:20.751039 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:20.751043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:20.751440 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:20.751590 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.751599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.752940 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.752944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.754006 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:20.754028 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.754706 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.754709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:20.754870 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.755430 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:20.755447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:20.758641 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
W20260929 03:02:20.759073 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:20.750167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:60293 (local address 127.2.205.190:35857)
0929 03:02:20.750351 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:20.750356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:20.750641 (+ 285us) server_negotiation.cc:407] Connection header received
0929 03:02:20.750773 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:20.750778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:20.750846 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:20.750949 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:20.751755 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.752814 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.754158 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:20.754600 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:20.754698 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:20.758831 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:60293: BlockingRecv error: recv got EOF from 127.2.205.130:60293 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:21.155244 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:21.146487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:58661)
0929 03:02:21.146987 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:21.147003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:21.147123 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:21.147407 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:21.147410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:21.147852 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:21.148001 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.148010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.149046 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.149050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.149958 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.149969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.150697 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.150703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.150878 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.151482 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:21.151501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:21.155013 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":255,"threads_started":1}
W20260929 03:02:21.155433 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:21.146624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:58661 (local address 127.2.205.190:35857)
0929 03:02:21.146794 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:21.146798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:21.147073 (+ 275us) server_negotiation.cc:407] Connection header received
0929 03:02:21.147211 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:21.147214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:21.147275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:21.147355 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:21.148129 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.148923 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.150067 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.150474 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.150638 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.155221 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:58661: BlockingRecv error: recv got EOF from 127.2.205.131:58661 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:21.275153 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:21.266114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:60575)
0929 03:02:21.266602 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:21.266618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:21.266737 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:21.266994 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:21.267008 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:21.267540 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:21.267732 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.267743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.268743 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.268774 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.269690 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.269702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.270348 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.270352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.270478 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.271040 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:21.271058 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:21.274902 (+ 3844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":239,"threads_started":1}
W20260929 03:02:21.275319 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:21.266193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:60575 (local address 127.2.205.190:35857)
0929 03:02:21.266356 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:21.266360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:21.266685 (+ 325us) server_negotiation.cc:407] Connection header received
0929 03:02:21.266819 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:21.266822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:21.266884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:21.266960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:21.267862 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.268633 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.269811 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.270234 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.270324 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.275111 (+ 4787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:60575: BlockingRecv error: recv got EOF from 127.2.205.129:60575 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:21 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:21.768328 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:21.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:47631)
0929 03:02:21.760438 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:21.760453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:21.760579 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:21.760996 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:21.761000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:21.761449 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:21.761619 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.761628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.762656 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.762661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.763597 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:21.763608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.764256 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.764260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:21.764391 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.764956 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:21.764973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:21.768097 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
W20260929 03:02:21.768440 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:21.760049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:47631 (local address 127.2.205.190:35857)
0929 03:02:21.760218 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:21.760224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:21.760533 (+ 309us) server_negotiation.cc:407] Connection header received
0929 03:02:21.760708 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:21.760713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:21.760840 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:21.760933 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:21.761738 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.762487 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.763715 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:21.764128 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:21.764214 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:21.768264 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:47631: BlockingRecv error: recv got EOF from 127.2.205.130:47631 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:22.165539 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:22.156312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:39411)
0929 03:02:22.156832 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:22.156847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:22.156985 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:22.157292 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:22.157297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:22.157772 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:22.157946 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.157956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.158960 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.158965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.160175 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.160190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.161068 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.161074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.161247 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.161826 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:22.161844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:22.165280 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
W20260929 03:02:22.165766 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:22.156482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:39411 (local address 127.2.205.190:35857)
0929 03:02:22.156717 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:22.156723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:22.156928 (+ 205us) server_negotiation.cc:407] Connection header received
0929 03:02:22.157110 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:22.157114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:22.157174 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:22.157248 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:22.158084 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.158818 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.160341 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.160899 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.161031 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.165527 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:39411: BlockingRecv error: recv got EOF from 127.2.205.131:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:22.285102 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:22.276083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:37629)
0929 03:02:22.276624 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:22.276641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:22.276784 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:22.277065 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:22.277068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:22.277439 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:22.277613 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.277622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.278669 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.278673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.279643 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.279656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.280326 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.280331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.280446 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.281028 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:22.281046 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:22.284835 (+ 3789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":245,"threads_started":1}
W20260929 03:02:22.285225 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:22.276158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:37629 (local address 127.2.205.190:35857)
0929 03:02:22.276307 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:22.276312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:22.276724 (+ 412us) server_negotiation.cc:407] Connection header received
0929 03:02:22.276876 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:22.276880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:22.276944 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:22.277024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:22.277744 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.278541 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.279780 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.280207 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.280301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.285045 (+ 4744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:37629: BlockingRecv error: recv got EOF from 127.2.205.129:37629 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:22 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:22.785727 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:22.769220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:39067)
0929 03:02:22.769792 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:22.769811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:22.769948 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:22.770282 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:22.770286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:22.770791 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:22.770964 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.770974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.771916 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.771920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.772867 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:22.772878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.773529 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.773534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:22.773638 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.781647 (+ 8009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:22.781678 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:22.785396 (+ 3718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":242,"threads_started":1}
W20260929 03:02:22.785858 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:22.769310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:39067 (local address 127.2.205.190:35857)
0929 03:02:22.769572 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:22.769577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:22.769891 (+ 314us) server_negotiation.cc:407] Connection header received
0929 03:02:22.770058 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:22.770063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:22.770144 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:22.770231 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:22.771094 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.771804 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.772980 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:22.773419 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:22.773517 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:22.785626 (+ 12109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:39067: BlockingRecv error: recv got EOF from 127.2.205.130:39067 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:23.175381 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:23.166469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:60261)
0929 03:02:23.167060 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:23.167078 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:23.167191 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:23.167551 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:23.167554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:23.168053 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:23.168239 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.168249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.169435 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.169440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.170407 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.170419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.171140 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.171146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.171270 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.171801 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:23.171818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:23.175039 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":271,"threads_started":1}
W20260929 03:02:23.175420 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:23.166556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:60261 (local address 127.2.205.190:35857)
0929 03:02:23.166718 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:23.166722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:23.167144 (+ 422us) server_negotiation.cc:407] Connection header received
0929 03:02:23.167320 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:23.167325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:23.167391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:23.167502 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:23.168377 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.169299 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.170549 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.170956 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.171052 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.175225 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:60261: BlockingRecv error: recv got EOF from 127.2.205.131:60261 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:23.296070 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:23.286214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:44609)
0929 03:02:23.286755 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:23.286772 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:23.286907 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:23.287271 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:23.287275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:23.287757 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:23.287964 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.287977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.289388 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.289395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.290553 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.290571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.291525 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.291531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.291686 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.292270 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:23.292288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:23.295726 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
W20260929 03:02:23.296204 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:23.286439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:44609 (local address 127.2.205.190:35857)
0929 03:02:23.286652 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:23.286657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:23.286889 (+ 232us) server_negotiation.cc:407] Connection header received
0929 03:02:23.287036 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:23.287040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:23.287114 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:23.287194 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:23.288125 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.289177 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.290725 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.291320 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.291436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.295971 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:44609: BlockingRecv error: recv got EOF from 127.2.205.129:44609 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:23 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:23.795418 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:23.786696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:37183)
0929 03:02:23.787267 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:23.787283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:23.787396 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:23.787709 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:23.787713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:23.788132 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:23.788297 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.788308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.789556 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.789560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.790496 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:23.790508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.791134 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.791139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:23.791249 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.791781 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:23.791797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:23.795177 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":324,"threads_started":1}
W20260929 03:02:23.795576 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:23.786801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:37183 (local address 127.2.205.190:35857)
0929 03:02:23.786991 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:23.786996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:23.787350 (+ 354us) server_negotiation.cc:407] Connection header received
0929 03:02:23.787482 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:23.787485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:23.787549 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:23.787635 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:23.788417 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.789394 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.790602 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:23.790996 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:23.791087 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:23.795382 (+ 4295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:37183: BlockingRecv error: recv got EOF from 127.2.205.130:37183 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:24.184801 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:24.176388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:59731)
0929 03:02:24.176951 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:24.176967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:24.177091 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:24.177403 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:24.177406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:24.177824 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:24.177993 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.178003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.179043 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.179046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.179968 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.179979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.180630 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.180634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.180743 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.181372 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:24.181390 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:24.184507 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
W20260929 03:02:24.184974 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:24.176469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:59731 (local address 127.2.205.190:35857)
0929 03:02:24.176665 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:24.176671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:24.177067 (+ 396us) server_negotiation.cc:407] Connection header received
0929 03:02:24.177186 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:24.177190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:24.177265 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:24.177342 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:24.178131 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.178922 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.180070 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.180473 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.180565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.184721 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:59731: BlockingRecv error: recv got EOF from 127.2.205.131:59731 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:24.305819 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:24.296980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:33541)
0929 03:02:24.297432 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:24.297447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:24.297578 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:24.297888 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:24.297893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:24.298307 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:24.298465 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.298475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.299483 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.299487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.300435 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.300447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.301199 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.301205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.301372 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.301938 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:24.301959 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:24.305504 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
W20260929 03:02:24.305867 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:24.297036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:33541 (local address 127.2.205.190:35857)
0929 03:02:24.297238 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:24.297244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:24.297566 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:02:24.297693 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:24.297698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:24.297767 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:24.297842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:24.298602 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.299345 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.300569 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.301052 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.301160 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.305676 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:33541: BlockingRecv error: recv got EOF from 127.2.205.129:33541 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:24 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:24.804986 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:24.796323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:60673)
0929 03:02:24.796858 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:24.796875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:24.796990 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:24.797300 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:24.797303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:24.797740 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:24.797891 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.797899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.798901 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.798905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.799886 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:24.799899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.800542 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.800547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:24.800672 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.801286 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:24.801310 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:24.804690 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
W20260929 03:02:24.805174 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:24.796334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:60673 (local address 127.2.205.190:35857)
0929 03:02:24.796486 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:24.796490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:24.796937 (+ 447us) server_negotiation.cc:407] Connection header received
0929 03:02:24.797076 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:24.797080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:24.797140 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:24.797219 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:24.798014 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.798765 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.799997 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:24.800390 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:24.800477 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:24.804928 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:60673: BlockingRecv error: recv got EOF from 127.2.205.130:60673 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:25.194970 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:25.185908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:43825)
0929 03:02:25.186368 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:25.186384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:25.186521 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:25.186827 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:25.186831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:25.187259 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:25.187400 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.187412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.188455 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.188460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.189479 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.189493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.190324 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.190329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.190471 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.191086 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:25.191105 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:25.194737 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
W20260929 03:02:25.195225 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:25.185957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:43825 (local address 127.2.205.190:35857)
0929 03:02:25.186193 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:25.186199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:25.186482 (+ 283us) server_negotiation.cc:407] Connection header received
0929 03:02:25.186640 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:25.186644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:25.186710 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:25.186785 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:25.187542 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.188290 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.189625 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.190098 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.190203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.194970 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:43825: BlockingRecv error: recv got EOF from 127.2.205.131:43825 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:25.315239 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:25.306876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:57069)
0929 03:02:25.307362 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:25.307381 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:25.307495 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:25.307764 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:25.307767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:25.308120 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:25.308281 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.308289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.309310 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.309314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.310267 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.310278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.310976 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.310980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.311096 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.311672 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:25.311690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:25.314983 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":254,"threads_started":1}
W20260929 03:02:25.315464 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:25.306937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:57069 (local address 127.2.205.190:35857)
0929 03:02:25.307113 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:25.307118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:25.307456 (+ 338us) server_negotiation.cc:407] Connection header received
0929 03:02:25.307595 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:25.307599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:25.307659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:25.307735 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:25.308436 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.309205 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.310427 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.310848 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.310958 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.315221 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:57069: BlockingRecv error: recv got EOF from 127.2.205.129:57069 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:25 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:25.814635 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:25.805964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:48285)
0929 03:02:25.806444 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:25.806460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:25.806583 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:25.806879 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:25.806883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:25.807259 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:25.807434 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.807443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.808403 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.808407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.809448 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:25.809462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.810072 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.810076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:25.810192 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.810762 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:25.810778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:25.814335 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260929 03:02:25.814881 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:25.806059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:48285 (local address 127.2.205.190:35857)
0929 03:02:25.806229 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:25.806233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:25.806526 (+ 293us) server_negotiation.cc:407] Connection header received
0929 03:02:25.806693 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:25.806696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:25.806758 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:25.806827 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:25.807566 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.808265 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.809564 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:25.809942 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:25.810032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:25.814605 (+ 4573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:48285: BlockingRecv error: recv got EOF from 127.2.205.130:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:26.204814 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:26.196179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:46379)
0929 03:02:26.196712 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:26.196730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:26.196879 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:26.197166 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:26.197169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:26.197562 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:26.197725 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.197733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.198707 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.198711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.199683 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.199693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.200310 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.200313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.200438 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.201015 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:26.201032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:26.204478 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
W20260929 03:02:26.204969 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:26.196412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:46379 (local address 127.2.205.190:35857)
0929 03:02:26.196644 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:26.196649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:26.196829 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:02:26.196978 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:26.196982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:26.197042 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:26.197136 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:26.197855 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.198602 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.199784 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.200205 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.200289 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.204673 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:46379: BlockingRecv error: recv got EOF from 127.2.205.131:46379 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:26.324867 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:26.316191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:39519)
0929 03:02:26.316607 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:26.316623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:26.316733 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:26.317030 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:26.317033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:26.317400 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:26.317575 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.317603 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.318563 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.318567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.319475 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.319487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.320226 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.320230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.320339 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.320959 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:26.320976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:26.324526 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":219,"threads_started":1}
W20260929 03:02:26.325016 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:26.316271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:39519 (local address 127.2.205.190:35857)
0929 03:02:26.316507 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:26.316513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:26.316690 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:02:26.316864 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:26.316868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:26.316928 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:26.316999 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:26.317746 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.318453 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.319619 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.320113 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.320209 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.324735 (+ 4526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:39519: BlockingRecv error: recv got EOF from 127.2.205.129:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:26 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:26.824862 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:26.815913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:40699)
0929 03:02:26.816386 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:26.816402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:26.816524 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:26.816827 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:26.816829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:26.817230 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:26.817412 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.817420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.818453 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.818457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.819370 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:26.819381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.820131 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.820136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:26.820266 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.820877 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:26.820896 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:26.824587 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
W20260929 03:02:26.825016 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:26.816030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:40699 (local address 127.2.205.190:35857)
0929 03:02:26.816252 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:26.816259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:26.816504 (+ 245us) server_negotiation.cc:407] Connection header received
0929 03:02:26.816627 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:26.816631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:26.816695 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:26.816813 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:26.817547 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.818336 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.819515 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:26.819997 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:26.820108 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:26.824790 (+ 4682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:40699: BlockingRecv error: recv got EOF from 127.2.205.130:40699 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:27.214689 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:27.205926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:50675)
0929 03:02:27.206587 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:27.206602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:27.206723 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:27.207000 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:27.207003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:27.207395 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:27.207615 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.207630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.208583 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.208588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.209601 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.209615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.210275 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.210280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.210402 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.210981 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:27.210999 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:27.214471 (+ 3472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":452,"threads_started":1}
W20260929 03:02:27.214860 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:27.206090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:50675 (local address 127.2.205.190:35857)
0929 03:02:27.206277 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:27.206284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:27.206686 (+ 402us) server_negotiation.cc:407] Connection header received
0929 03:02:27.206814 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:27.206818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:27.206882 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:27.206967 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:27.207739 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.208433 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.209743 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.210156 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.210249 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.214658 (+ 4409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:50675: BlockingRecv error: recv got EOF from 127.2.205.131:50675 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:27.334901 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:27.325895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:47125)
0929 03:02:27.326417 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:27.326433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:27.326566 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:27.326881 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:27.326885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:27.327322 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:27.327493 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.327503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.328519 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.328525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.329630 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.329642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.330378 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.330385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.330568 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.331132 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:27.331150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:27.334583 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
W20260929 03:02:27.335103 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:27.326063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:47125 (local address 127.2.205.190:35857)
0929 03:02:27.326324 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:27.326330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:27.326510 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:02:27.326684 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:27.326688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:27.326750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:27.326829 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:27.327639 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.328379 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.329770 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.330202 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.330312 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.334825 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:47125: BlockingRecv error: recv got EOF from 127.2.205.129:47125 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:27 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:27.840787 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:27.825994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:36667)
0929 03:02:27.826530 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:27.826548 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:27.826707 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:27.827061 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:27.827066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:27.827511 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:27.827689 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.827699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.828675 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.828679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.829618 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:27.829630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.830334 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.830341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:27.830495 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.837296 (+ 6801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:27.837315 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:27.840400 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":280,"threads_started":1}
W20260929 03:02:27.840878 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:27.826060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:36667 (local address 127.2.205.190:35857)
0929 03:02:27.826251 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:27.826258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:27.826643 (+ 385us) server_negotiation.cc:407] Connection header received
0929 03:02:27.826828 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:27.826833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:27.826906 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:27.826990 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:27.827833 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.828545 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.829754 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:27.830166 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:27.830273 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:27.840614 (+ 10341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:36667: BlockingRecv error: recv got EOF from 127.2.205.130:36667 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:28.225613 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:28.215606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:36567)
0929 03:02:28.216152 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:28.216169 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:28.216292 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:28.216668 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:28.216672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:28.217134 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:28.217310 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.217321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.218441 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.218445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.219440 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.219453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.220249 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.220254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.220388 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.221235 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:28.221281 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:28.225231 (+ 3950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":284,"threads_started":1}
W20260929 03:02:28.225744 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:28.215743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:36567 (local address 127.2.205.190:35857)
0929 03:02:28.215917 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:28.215921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:28.216261 (+ 340us) server_negotiation.cc:407] Connection header received
0929 03:02:28.216439 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:28.216444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:28.216510 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:28.216610 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:28.217458 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.218251 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.219606 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.220050 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.220157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.225492 (+ 5335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:36567: BlockingRecv error: recv got EOF from 127.2.205.131:36567 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:28.344424 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:28.335884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:59211)
0929 03:02:28.336383 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:28.336399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:28.336517 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:28.336842 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:28.336845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:28.337240 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:28.337408 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.337418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.338360 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.338364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.339317 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.339329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.340023 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.340027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.340168 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.340822 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:28.340841 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:28.344135 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
W20260929 03:02:28.344578 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:28.335936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:59211 (local address 127.2.205.190:35857)
0929 03:02:28.336114 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:28.336118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:28.336462 (+ 344us) server_negotiation.cc:407] Connection header received
0929 03:02:28.336641 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:28.336644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:28.336706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:28.336822 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:28.337527 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.338234 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.339467 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.339902 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.340002 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.344366 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:59211: BlockingRecv error: recv got EOF from 127.2.205.129:59211 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:28 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:28.849908 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:28.841721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:44305)
0929 03:02:28.842181 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:28.842196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:28.842324 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:28.842635 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:28.842640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:28.843059 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:28.843202 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.843211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.844194 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.844199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.845157 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:28.845170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.845868 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.845874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:28.846038 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.846572 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:28.846589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:28.849594 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
W20260929 03:02:28.850016 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:28.841908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:44305 (local address 127.2.205.190:35857)
0929 03:02:28.842106 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:28.842112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:28.842283 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:02:28.842453 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:28.842456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:28.842516 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:28.842593 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:28.843343 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.844062 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.845293 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:28.845737 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:28.845843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:28.849795 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:44305: BlockingRecv error: recv got EOF from 127.2.205.130:44305 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.131@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:29.235746 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.226679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.131:56283)
0929 03:02:29.227115 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:29.227131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:29.227249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:29.227652 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:29.227656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:29.228046 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:29.228193 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.228202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.229241 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.229245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.230180 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.230193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.230889 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.230895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.231059 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.231639 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:29.231659 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:29.235431 (+ 3772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
W20260929 03:02:29.235870 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:29.226788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:56283 (local address 127.2.205.190:35857)
0929 03:02:29.226998 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.227004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.227202 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:02:29.227387 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.227421 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.227493 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.227584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.228329 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.229139 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.230330 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.230751 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.230858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.235634 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.131:56283: BlockingRecv error: recv got EOF from 127.2.205.131:56283 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.129@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:29.353977 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.345843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.129:35111)
0929 03:02:29.346271 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:29.346286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:29.346391 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:29.346709 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:29.346712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:29.347066 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:29.347240 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.347248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.348194 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.348197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.349201 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.349213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.349866 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.349870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.350008 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.350575 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:29.350591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:29.353724 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":238,"threads_started":1}
W20260929 03:02:29.354140 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:29.345865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:35111 (local address 127.2.205.190:35857)
0929 03:02:29.346160 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.346166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.346356 (+ 190us) server_negotiation.cc:407] Connection header received
0929 03:02:29.346532 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.346536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.346602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.346684 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.347352 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.348076 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.349324 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.349758 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.349852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.353892 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.129:35111: BlockingRecv error: recv got EOF from 127.2.205.129:35111 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:02:29 dist-test-slave-530n krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.205.130@KRBTEST.COM for oryx/127.2.205.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:02:29.859705 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.851118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35857 (local address 127.2.205.130:36135)
0929 03:02:29.851618 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:29.851634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:29.851765 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:29.852066 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:29.852070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:29.852464 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:29.852655 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.852664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.853671 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.853676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.854959 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:29.854975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.855645 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.855650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:29.855821 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.856352 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:29.856370 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:29.859451 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.205.190:35857: Server oryx/127.2.205.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":220,"threads_started":1}
W20260929 03:02:29.859871 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:29.851294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:36135 (local address 127.2.205.190:35857)
0929 03:02:29.851517 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.851524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.851716 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:02:29.851882 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.851886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.851948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.852024 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.852819 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.853529 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.855099 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.855505 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.855613 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.859664 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.205.130:36135: BlockingRecv error: recv got EOF from 127.2.205.130:36135 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 03:02:29.901564 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11717
I20260929 03:02:29.914240 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.902862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.2.205.129:37049)
0929 03:02:29.903452 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.903460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.903487 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:02:29.903551 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.903558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.903788 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.903955 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.905428 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.906505 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.907813 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.908400 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.908512 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.911005 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:29.911014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:29.911030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:29.911081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:29.913702 (+ 2621us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 03:02:29.914008 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41292: Request ticket server kudu/127.2.205.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":297,"threads_started":1}
W20260929 03:02:29.914443 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11717: Not authorized: Client connection negotiation failed: client connection to 127.2.205.129:37049: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 03:02:29.914642 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11717
I20260929 03:02:29.928571 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11868
I20260929 03:02:29.941313 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.929672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.2.205.130:38077)
0929 03:02:29.930331 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.930340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.930377 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:02:29.930459 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.930468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.930783 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.931025 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.932258 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.933668 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.934896 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.935503 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.935610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.938212 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:29.938222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:29.938237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:29.938282 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:29.940701 (+ 2419us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 03:02:29.941048 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36286: Request ticket server kudu/127.2.205.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":291,"threads_started":1}
W20260929 03:02:29.941426 12181 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36286: Request ticket server kudu/127.2.205.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 03:02:29.941499 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11868: Not authorized: Client connection negotiation failed: client connection to 127.2.205.130:38077: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 03:02:29.941659 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11868
I20260929 03:02:29.955698 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12001
I20260929 03:02:29.967757 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:29.956717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.2.205.131:42417)
0929 03:02:29.957315 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:29.957322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:29.957348 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:02:29.957418 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:29.957424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:29.957658 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:29.957816 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:29.959498 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.960509 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.961788 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:29.962404 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:29.962512 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:29.964805 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:29.964815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:29.964829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:29.964866 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:29.967266 (+ 2400us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 03:02:29.967530 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53514: Request ticket server kudu/127.2.205.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":271,"threads_started":1}
W20260929 03:02:29.967850 12182 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53514: Request ticket server kudu/127.2.205.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 03:02:29.967868 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12001: Not authorized: Client connection negotiation failed: client connection to 127.2.205.131:42417: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.205.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 03:02:29.968029 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12001
I20260929 03:02:29.981004 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 11648
I20260929 03:02:29.982360 11710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:30.097319 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 11648
2026-09-29T03:02:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20761 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:30Z Disabled control of system clock
I20260929 03:02:30.166182 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35929
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:02:30.490280 12187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:30.490586 12187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:30.490682 12187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:30.497903 12187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:30.498006 12187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:30.498051 12187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:30.498106 12187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:30.498149 12187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:30.507149 12187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:30.508394 12187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:30.510083 12187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:30.521130 12192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:30.521102 12193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:30.521531 12195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:30.521799 12187 server_base.cc:1034] running on GCE node
I20260929 03:02:30.522608 12187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:30.524166 12187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:30.525390 12187 hybrid_clock.cc:648] HybridClock initialized: now 1790650950525329 us; error 53 us; skew 500 ppm
I20260929 03:02:30.525775 12187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:30.528092 12187 webserver.cc:460] Webserver started at http://127.2.205.190:37359/ using document root <none> and password file <none>
I20260929 03:02:30.528822 12187 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:30.528930 12187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:30.529256 12187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:30.532401 12187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "3c4f364ee92a4196a1f333a7ca362dbb"
format_stamp: "Formatted at 2026-09-29 03:02:30 on dist-test-slave-530n"
server_key: "f4bd66f3e3d274800802dcd0bccc35ad"
server_key_iv: "1d8d4f04238f3b46d8ab874448b588f0"
server_key_version: "encryptionkey@0"
I20260929 03:02:30.533322 12187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "3c4f364ee92a4196a1f333a7ca362dbb"
format_stamp: "Formatted at 2026-09-29 03:02:30 on dist-test-slave-530n"
server_key: "f4bd66f3e3d274800802dcd0bccc35ad"
server_key_iv: "1d8d4f04238f3b46d8ab874448b588f0"
server_key_version: "encryptionkey@0"
I20260929 03:02:30.538954 12187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:02:30.542966 12201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:30.544476 12187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:02:30.544657 12187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "3c4f364ee92a4196a1f333a7ca362dbb"
format_stamp: "Formatted at 2026-09-29 03:02:30 on dist-test-slave-530n"
server_key: "f4bd66f3e3d274800802dcd0bccc35ad"
server_key_iv: "1d8d4f04238f3b46d8ab874448b588f0"
server_key_version: "encryptionkey@0"
I20260929 03:02:30.544932 12187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:30.557423 12187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:30.558298 12187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:30.558583 12187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:30.575990 12187 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35929
I20260929 03:02:30.576010 12252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35929 every 8 connection(s)
I20260929 03:02:30.577419 12187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:30.581586 12253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:30.585194 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12187
I20260929 03:02:30.585400 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:02:30.586202 2870 external_mini_cluster.cc:1428] Setting key de974cd9c9f85eaa2228f6fa96e61f87
I20260929 03:02:30.594949 12253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb: Bootstrap starting.
I20260929 03:02:30.598351 12253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:30.599534 12253 log.cc:824] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:30.600571 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:30.588564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.2.205.190:35929)
0929 03:02:30.589661 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:30.589675 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:30.589720 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:02:30.591385 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:30.591413 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:30.591982 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:30.592438 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:30.593119 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:30.594383 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:30.595758 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:30.596502 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:30.596686 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:30.597966 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:30.597975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:30.597994 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:30.598627 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:30.598745 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:30.598773 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:30.598887 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:30.599107 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 03:02:30.599439 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":389,"threads_started":1}
I20260929 03:02:30.602648 12253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb: No bootstrap required, opened a new log
I20260929 03:02:30.609791 12253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } }
I20260929 03:02:30.610175 12253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:30.610278 12253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4f364ee92a4196a1f333a7ca362dbb, State: Initialized, Role: FOLLOWER
I20260929 03:02:30.610841 12253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } }
I20260929 03:02:30.611055 12253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:30.611178 12253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:30.611321 12253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:30.613638 12253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } }
I20260929 03:02:30.614145 12253 leader_election.cc:302] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c4f364ee92a4196a1f333a7ca362dbb; no voters:
I20260929 03:02:30.614688 12253 leader_election.cc:288] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:02:30.614831 12258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:30.615530 12258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [term 1 LEADER]: Becoming Leader. State: Replica: 3c4f364ee92a4196a1f333a7ca362dbb, State: Running, Role: LEADER
I20260929 03:02:30.616086 12258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } }
I20260929 03:02:30.616700 12253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:30.619230 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } } }
I20260929 03:02:30.619078 12260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c4f364ee92a4196a1f333a7ca362dbb. Latest consensus state: current_term: 1 leader_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4f364ee92a4196a1f333a7ca362dbb" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35929 } } }
I20260929 03:02:30.619671 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:30.619727 12260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:30.620448 12264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:30.626142 12264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:30.636616 12264 catalog_manager.cc:1347] Generated new cluster ID: 10eb7ba42a9d4ba1ab87fd7ba4191b10
I20260929 03:02:30.636770 12264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:30.654053 12264 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:02:30.655356 12264 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:30.664083 12264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c4f364ee92a4196a1f333a7ca362dbb: Generated new TSK 0
I20260929 03:02:30.665117 12264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:02:30.675355 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--builtin_ntp_servers=127.2.205.148:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:02:31.002800 12277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:31.003124 12277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:31.003254 12277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:31.010612 12277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:31.010741 12277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:31.010887 12277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:31.020324 12277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:31.021687 12277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:31.023550 12277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:31.037093 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.037057 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.037124 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:31.037842 12277 server_base.cc:1034] running on GCE node
I20260929 03:02:31.038506 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:31.039520 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:31.040732 12277 hybrid_clock.cc:648] HybridClock initialized: now 1790650951040665 us; error 55 us; skew 500 ppm
I20260929 03:02:31.041093 12277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:31.043371 12277 webserver.cc:460] Webserver started at http://127.2.205.129:38723/ using document root <none> and password file <none>
I20260929 03:02:31.044085 12277 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:31.044178 12277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:31.044473 12277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:31.047468 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "c5817a6cf4c24ca78671147cf143b028"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "207502e6ca5efb1601aa6c08e60f3240"
server_key_iv: "760cdbbf5f6a401e322cc4341c9df286"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.048200 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "c5817a6cf4c24ca78671147cf143b028"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "207502e6ca5efb1601aa6c08e60f3240"
server_key_iv: "760cdbbf5f6a401e322cc4341c9df286"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.053378 12277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 03:02:31.056712 12291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.058013 12277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:02:31.058142 12277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "c5817a6cf4c24ca78671147cf143b028"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "207502e6ca5efb1601aa6c08e60f3240"
server_key_iv: "760cdbbf5f6a401e322cc4341c9df286"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.058327 12277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:31.074630 12277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:31.075501 12277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:31.075762 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:31.076623 12277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:31.078008 12277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:31.078099 12277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.078202 12277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:31.078275 12277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.107589 12277 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:35669
I20260929 03:02:31.107627 12403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:35669 every 8 connection(s)
I20260929 03:02:31.108986 12277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:31.116458 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12277
I20260929 03:02:31.116637 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:02:31.117389 2870 external_mini_cluster.cc:1428] Setting key 0a5f28cce074d13c2b804622cc25186a
I20260929 03:02:31.120283 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--builtin_ntp_servers=127.2.205.148:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:02:31.122291 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:31.111632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:36805 (local address 127.2.205.190:35929)
0929 03:02:31.112049 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:31.112053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:31.113405 (+ 1352us) server_negotiation.cc:407] Connection header received
0929 03:02:31.115201 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:31.115207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:31.115290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:31.115383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:31.116533 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.117424 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.119174 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.119763 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.119896 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:31.121442 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:31.121449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:31.121454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:31.121477 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:31.121538 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:31.121562 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:31.121653 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:31.121845 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:02:31.121952 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260929 03:02:31.123291 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:31.112281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35929 (local address 127.2.205.129:36805)
0929 03:02:31.113235 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:31.113280 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:31.114911 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:31.115640 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:31.115649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:31.115746 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:31.116357 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:31.116375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.117583 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.117587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:31.119041 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:31.119057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.120048 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.120057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:31.120321 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:31.121104 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:31.121107 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:31.121153 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:31.121242 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:31.121652 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:31.121660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:31.121665 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:02:31.121822 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 03:02:31.122170 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":274,"threads_started":1}
I20260929 03:02:31.124800 12404 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35929
I20260929 03:02:31.125131 12404 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:31.126140 12404 heartbeater.cc:511] Master 127.2.205.190:35929 requested a full tablet report, sending...
I20260929 03:02:31.128351 12218 ts_manager.cc:194] Registered new tserver with Master: c5817a6cf4c24ca78671147cf143b028 (127.2.205.129:35669)
I20260929 03:02:31.129897 12218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.129:36805
W20260929 03:02:31.447682 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:31.448004 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:31.448130 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:31.455461 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:31.455578 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:31.455729 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:31.464850 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:31.466145 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:31.467834 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:31.480031 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.479979 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.480002 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:31.480566 12408 server_base.cc:1034] running on GCE node
I20260929 03:02:31.481381 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:31.482383 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:31.483608 12408 hybrid_clock.cc:648] HybridClock initialized: now 1790650951483560 us; error 49 us; skew 500 ppm
I20260929 03:02:31.483937 12408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:31.486326 12408 webserver.cc:460] Webserver started at http://127.2.205.130:33487/ using document root <none> and password file <none>
I20260929 03:02:31.487046 12408 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:31.487142 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:31.487462 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:31.490469 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "8e9ed928913646e7ab62a3e718a102e5"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "f27e907ee5a32a4322415e72560bd19a"
server_key_iv: "2ca89b64b8e139b08b9199cd65975e93"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.491283 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "8e9ed928913646e7ab62a3e718a102e5"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "f27e907ee5a32a4322415e72560bd19a"
server_key_iv: "2ca89b64b8e139b08b9199cd65975e93"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.497020 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 03:02:31.500906 12422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.502481 12408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:02:31.502672 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "8e9ed928913646e7ab62a3e718a102e5"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "f27e907ee5a32a4322415e72560bd19a"
server_key_iv: "2ca89b64b8e139b08b9199cd65975e93"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.502899 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:31.525624 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:31.526573 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:31.526862 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:31.528017 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:31.529661 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:31.529759 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.529866 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:31.529932 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.558972 12408 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:46701
I20260929 03:02:31.559041 12534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:46701 every 8 connection(s)
I20260929 03:02:31.560215 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:31.563555 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12408
I20260929 03:02:31.563691 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:02:31.564122 2870 external_mini_cluster.cc:1428] Setting key d854ba54cf890069086b74587c21fbb0
I20260929 03:02:31.566773 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--builtin_ntp_servers=127.2.205.148:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:02:31.573029 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:31.562865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:39423 (local address 127.2.205.190:35929)
0929 03:02:31.563017 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:31.563021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:31.564587 (+ 1566us) server_negotiation.cc:407] Connection header received
0929 03:02:31.566679 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:31.566684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:31.566762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:31.566870 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:31.567910 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.568858 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.570354 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.570904 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.571017 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:31.572280 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:31.572286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:31.572291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:31.572312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:31.572351 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:31.572365 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:31.572446 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:31.572662 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:02:31.572785 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:02:31.574340 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:31.563381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35929 (local address 127.2.205.130:39423)
0929 03:02:31.564424 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:31.564468 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:31.566429 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:31.567104 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:31.567113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:31.567200 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:31.567756 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:31.567771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.568998 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.569002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:31.570231 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:31.570243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:31.571101 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:31.571106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:31.571306 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:31.571974 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:31.571977 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:31.572015 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:31.572100 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:31.572461 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:31.572469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:31.572476 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:02:31.572638 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 03:02:31.573065 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":328,"threads_started":1}
I20260929 03:02:31.576092 12535 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35929
I20260929 03:02:31.576452 12535 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:31.577360 12535 heartbeater.cc:511] Master 127.2.205.190:35929 requested a full tablet report, sending...
I20260929 03:02:31.578922 12218 ts_manager.cc:194] Registered new tserver with Master: 8e9ed928913646e7ab62a3e718a102e5 (127.2.205.130:46701)
I20260929 03:02:31.580050 12218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.130:39423
W20260929 03:02:31.891999 12538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:31.892318 12538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:31.892441 12538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:31.899781 12538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:31.899917 12538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:31.900065 12538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:31.909636 12538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:31.911020 12538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:31.912866 12538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:31.926360 12544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.926430 12546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:31.926389 12543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:31.927126 12538 server_base.cc:1034] running on GCE node
I20260929 03:02:31.927744 12538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:31.928773 12538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:31.929991 12538 hybrid_clock.cc:648] HybridClock initialized: now 1790650951929942 us; error 41 us; skew 500 ppm
I20260929 03:02:31.930342 12538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:31.932667 12538 webserver.cc:460] Webserver started at http://127.2.205.131:43469/ using document root <none> and password file <none>
I20260929 03:02:31.933429 12538 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:31.933524 12538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:31.933833 12538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:31.936789 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "eae916ae16444e09b71eb55bf7521247"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "8998d9fc9955dc974b48506575c6088b"
server_key_iv: "1b3b4961e2972b8bb0c716143c820774"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.937640 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "eae916ae16444e09b71eb55bf7521247"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "8998d9fc9955dc974b48506575c6088b"
server_key_iv: "1b3b4961e2972b8bb0c716143c820774"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.942955 12538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:02:31.946475 12552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.947829 12538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:02:31.947979 12538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "eae916ae16444e09b71eb55bf7521247"
format_stamp: "Formatted at 2026-09-29 03:02:31 on dist-test-slave-530n"
server_key: "8998d9fc9955dc974b48506575c6088b"
server_key_iv: "1b3b4961e2972b8bb0c716143c820774"
server_key_version: "encryptionkey@0"
I20260929 03:02:31.948186 12538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:31.963649 12538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:31.964428 12538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:31.964694 12538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:31.965576 12538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:31.966944 12538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:31.967034 12538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.967140 12538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:31.967202 12538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:31.996919 12538 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:40033
I20260929 03:02:31.996974 12664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:40033 every 8 connection(s)
I20260929 03:02:31.998200 12538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:32.000697 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12538
I20260929 03:02:32.000906 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:02:32.001333 2870 external_mini_cluster.cc:1428] Setting key a3b2f3d6b37ff6bd61627a4f5fec22a1
I20260929 03:02:32.010399 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.000651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:52127 (local address 127.2.205.190:35929)
0929 03:02:32.000893 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.000899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.002434 (+ 1535us) server_negotiation.cc:407] Connection header received
0929 03:02:32.004461 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.004466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.004536 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.004618 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.005746 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.006497 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.008079 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.008479 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.008577 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:32.009774 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:32.009779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:32.009782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:32.009798 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:32.009829 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:32.009840 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:32.009904 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:32.010090 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:02:32.010165 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:02:32.011351 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.001240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35929 (local address 127.2.205.131:52127)
0929 03:02:32.002259 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:32.002310 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:32.004191 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:32.004931 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:32.004939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:32.005031 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:32.005595 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:32.005622 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.006627 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.006631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:32.007931 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:32.007943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.008660 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.008665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:32.008900 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:32.009483 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:32.009485 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:32.009523 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:32.009598 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:32.009914 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:32.009920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:32.009925 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:02:32.010057 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:02:32.010344 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":288,"threads_started":1}
I20260929 03:02:32.012583 12665 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35929
I20260929 03:02:32.012868 12665 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:32.013589 12665 heartbeater.cc:511] Master 127.2.205.190:35929 requested a full tablet report, sending...
I20260929 03:02:32.014832 12218 ts_manager.cc:194] Registered new tserver with Master: eae916ae16444e09b71eb55bf7521247 (127.2.205.131:40033)
I20260929 03:02:32.015676 12218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.131:52127
I20260929 03:02:32.016590 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:02:32.023272 12406 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:32.021587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.2.205.190:35929)
0929 03:02:32.021780 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.021786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.022155 (+ 369us) server_negotiation.cc:407] Connection header received
0929 03:02:32.022324 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.022330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.022431 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.022516 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.022999 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53326: BlockingRecv error: recv got EOF from 127.0.0.1:53326 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:02:32.024061 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12277
I20260929 03:02:32.031667 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.024932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.2.205.129:35669)
0929 03:02:32.025409 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.025416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.025443 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:02:32.025522 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.025530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.025777 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.025917 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.026263 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.027362 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.028620 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.029268 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.029374 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:32.030369 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:32.030377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:32.030389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:32.030976 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:32.031025 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:32.031044 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:32.031136 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:32.031230 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:02:32.031359 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":238,"threads_started":1}
I20260929 03:02:32.033013 12399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:32.114197 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12277
I20260929 03:02:32.128067 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12408
I20260929 03:02:32.135537 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.128982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.2.205.130:46701)
0929 03:02:32.129488 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.129496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.129525 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:02:32.129596 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.129603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.129825 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.129963 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.130318 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.131305 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.132464 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.133139 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.133233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:32.134202 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:32.134212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:32.134226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:32.134831 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:32.134886 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:32.134904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:32.134991 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:32.135120 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:02:32.135249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
I20260929 03:02:32.136823 12530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:32.215646 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12408
I20260929 03:02:32.228986 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12538
I20260929 03:02:32.237773 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.230110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.2.205.131:40033)
0929 03:02:32.230875 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.230882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.230915 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:02:32.230983 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.230990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.231235 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.231392 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.231874 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.232995 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.234345 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:32.234927 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:32.235057 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:32.236166 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:32.236174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:32.236189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:32.236911 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:32.236969 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:32.236998 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:32.237101 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:32.237258 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:02:32.237417 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":407,"threads_started":1}
I20260929 03:02:32.239113 12660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:32.320073 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12538
I20260929 03:02:32.333389 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12187
I20260929 03:02:32.334664 12248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:32.436017 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12187
2026-09-29T03:02:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2333 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:32Z Disabled control of system clock
I20260929 03:02:32.501731 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45099
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:02:32.823808 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:32.824090 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:32.824196 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:32.831408 12684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:32.831506 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:32.831558 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:32.831634 12684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:32.831691 12684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:32.841024 12684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:32.842398 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:32.844246 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:32.855575 12692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:32.855501 12689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:32.855659 12690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:32.856660 12684 server_base.cc:1034] running on GCE node
I20260929 03:02:32.857476 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:32.858840 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:32.860052 12684 hybrid_clock.cc:648] HybridClock initialized: now 1790650952860002 us; error 46 us; skew 500 ppm
I20260929 03:02:32.860383 12684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:32.867348 12684 webserver.cc:460] Webserver started at http://127.2.205.190:43363/ using document root <none> and password file <none>
I20260929 03:02:32.868091 12684 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:32.868183 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:32.868500 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:32.871781 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "ce897805f47145a3881f35c6634dc0c8"
format_stamp: "Formatted at 2026-09-29 03:02:32 on dist-test-slave-530n"
server_key: "460620ecbfc95e7ba07092370757b6a1"
server_key_iv: "61499ed4849709670438841392910763"
server_key_version: "encryptionkey@0"
I20260929 03:02:32.872682 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "ce897805f47145a3881f35c6634dc0c8"
format_stamp: "Formatted at 2026-09-29 03:02:32 on dist-test-slave-530n"
server_key: "460620ecbfc95e7ba07092370757b6a1"
server_key_iv: "61499ed4849709670438841392910763"
server_key_version: "encryptionkey@0"
I20260929 03:02:32.878156 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 03:02:32.881922 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:32.883577 12684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:02:32.883749 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "ce897805f47145a3881f35c6634dc0c8"
format_stamp: "Formatted at 2026-09-29 03:02:32 on dist-test-slave-530n"
server_key: "460620ecbfc95e7ba07092370757b6a1"
server_key_iv: "61499ed4849709670438841392910763"
server_key_version: "encryptionkey@0"
I20260929 03:02:32.883962 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:32.890043 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:32.890938 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:32.891237 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:32.909360 12684 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:45099
I20260929 03:02:32.909391 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:45099 every 8 connection(s)
I20260929 03:02:32.910813 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:32.913650 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12684
I20260929 03:02:32.913853 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:02:32.914441 2870 external_mini_cluster.cc:1428] Setting key 6c2c0ac695e374518a5ab81d2d7d9c8b
I20260929 03:02:32.914788 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:32.923804 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:32.916444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.2.205.190:45099)
0929 03:02:32.917510 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:32.917524 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:32.917587 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:02:32.919593 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:32.919623 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:32.920208 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:32.920686 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:32.921746 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:32.921755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:32.921776 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:32.921793 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:32.921870 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:32.921893 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:32.922116 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:32.922250 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:02:32.922572 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":318,"threads_started":1}
I20260929 03:02:32.928848 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8: Bootstrap starting.
I20260929 03:02:32.931895 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:32.932962 12750 log.cc:824] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:32.935873 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8: No bootstrap required, opened a new log
I20260929 03:02:32.943581 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } }
I20260929 03:02:32.944051 12750 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:32.944155 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce897805f47145a3881f35c6634dc0c8, State: Initialized, Role: FOLLOWER
I20260929 03:02:32.944803 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } }
I20260929 03:02:32.945056 12750 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:32.945185 12750 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:32.945340 12750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:32.947582 12750 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } }
I20260929 03:02:32.948118 12750 leader_election.cc:302] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce897805f47145a3881f35c6634dc0c8; no voters:
I20260929 03:02:32.948725 12750 leader_election.cc:288] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:02:32.948899 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:32.950016 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [term 1 LEADER]: Becoming Leader. State: Replica: ce897805f47145a3881f35c6634dc0c8, State: Running, Role: LEADER
I20260929 03:02:32.950614 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } }
I20260929 03:02:32.951100 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:32.953444 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce897805f47145a3881f35c6634dc0c8. Latest consensus state: current_term: 1 leader_uuid: "ce897805f47145a3881f35c6634dc0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } } }
I20260929 03:02:32.953840 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:32.954265 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce897805f47145a3881f35c6634dc0c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce897805f47145a3881f35c6634dc0c8" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45099 } } }
I20260929 03:02:32.954563 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:32.961802 12763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:32.966255 12763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:32.974395 12763 catalog_manager.cc:1347] Generated new cluster ID: 4792754f030b41b0b3220fb0f55bbb31
I20260929 03:02:32.974512 12763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:32.998809 12763 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:02:32.999279 12763 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:33.008093 12763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce897805f47145a3881f35c6634dc0c8: Generated new TSK 0
I20260929 03:02:33.009127 12763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:02:33.023888 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--builtin_ntp_servers=127.2.205.148:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:02:33.363523 12774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:33.363790 12774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:33.363916 12774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:33.371116 12774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:33.371225 12774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:33.371357 12774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:33.380272 12774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:33.381697 12774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:33.383651 12774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:33.397485 12780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:33.397507 12782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:33.397694 12779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:33.398388 12774 server_base.cc:1034] running on GCE node
I20260929 03:02:33.399026 12774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:33.400063 12774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:33.401268 12774 hybrid_clock.cc:648] HybridClock initialized: now 1790650953401215 us; error 51 us; skew 500 ppm
I20260929 03:02:33.401607 12774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:33.404127 12774 webserver.cc:460] Webserver started at http://127.2.205.129:42513/ using document root <none> and password file <none>
I20260929 03:02:33.404942 12774 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:33.405061 12774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:33.405383 12774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:33.408377 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "19222047f01345309d09a369004b2164"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "8313e30bd484b92cc55e84cbaef4f209"
server_key_iv: "b62bb93a3da9d59d1cf9ff949a896652"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.409205 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "19222047f01345309d09a369004b2164"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "8313e30bd484b92cc55e84cbaef4f209"
server_key_iv: "b62bb93a3da9d59d1cf9ff949a896652"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.414336 12774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:02:33.417801 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.419154 12774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:33.419301 12774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "19222047f01345309d09a369004b2164"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "8313e30bd484b92cc55e84cbaef4f209"
server_key_iv: "b62bb93a3da9d59d1cf9ff949a896652"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.419508 12774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:33.425068 12774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:33.425851 12774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:33.426093 12774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:33.427021 12774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:33.428423 12774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:33.428524 12774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.428629 12774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:33.428696 12774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.458173 12774 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:36441
I20260929 03:02:33.458240 12900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:36441 every 8 connection(s)
I20260929 03:02:33.459420 12774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:33.467163 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:33.461853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:33425 (local address 127.2.205.190:45099)
0929 03:02:33.462220 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:33.462224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:33.463397 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 03:02:33.465054 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:33.465060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:33.465127 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:33.465220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:33.466424 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:33.466430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:33.466435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:33.466454 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:33.466499 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:33.466511 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:33.466607 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:33.466792 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:02:33.466856 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":188,"threads_started":1}
I20260929 03:02:33.467669 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12774
I20260929 03:02:33.467844 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:02:33.468448 2870 external_mini_cluster.cc:1428] Setting key a939c921feae9306ef74aee184ded823
I20260929 03:02:33.468780 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:33.462370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45099 (local address 127.2.205.129:33425)
0929 03:02:33.463249 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:33.463289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:33.464801 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:33.465452 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:33.465460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:33.465552 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:33.466146 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:33.466149 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:33.466185 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:33.466266 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:33.466593 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:33.466603 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:33.466610 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:02:33.466860 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 03:02:33.467115 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":247,"threads_started":1}
I20260929 03:02:33.470501 12901 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45099
I20260929 03:02:33.470777 12901 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:33.471289 12901 heartbeater.cc:511] Master 127.2.205.190:45099 requested a full tablet report, sending...
I20260929 03:02:33.471312 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--builtin_ntp_servers=127.2.205.148:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:02:33.473229 12715 ts_manager.cc:194] Registered new tserver with Master: 19222047f01345309d09a369004b2164 (127.2.205.129:36441)
W20260929 03:02:33.796489 12905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:33.796854 12905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:33.796973 12905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:33.804340 12905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:33.804451 12905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:33.804589 12905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:33.813728 12905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:33.814975 12905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:33.816659 12905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:33.829160 12910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:33.829173 12911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:33.829185 12913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:33.830258 12905 server_base.cc:1034] running on GCE node
I20260929 03:02:33.830794 12905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:33.831709 12905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:33.832870 12905 hybrid_clock.cc:648] HybridClock initialized: now 1790650953832825 us; error 29 us; skew 500 ppm
I20260929 03:02:33.833176 12905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:33.835225 12905 webserver.cc:460] Webserver started at http://127.2.205.130:38903/ using document root <none> and password file <none>
I20260929 03:02:33.835843 12905 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:33.835945 12905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:33.836256 12905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:33.838967 12905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "0e340c6acaa94107a6759d35d65f4914"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "dc17e52a7e7c3ccfa00c232cef4277ef"
server_key_iv: "e444370d4a07f26aa2d510321ec8bfa8"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.839736 12905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "0e340c6acaa94107a6759d35d65f4914"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "dc17e52a7e7c3ccfa00c232cef4277ef"
server_key_iv: "e444370d4a07f26aa2d510321ec8bfa8"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.844945 12905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 03:02:33.848210 12919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.849545 12905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:33.849699 12905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "0e340c6acaa94107a6759d35d65f4914"
format_stamp: "Formatted at 2026-09-29 03:02:33 on dist-test-slave-530n"
server_key: "dc17e52a7e7c3ccfa00c232cef4277ef"
server_key_iv: "e444370d4a07f26aa2d510321ec8bfa8"
server_key_version: "encryptionkey@0"
I20260929 03:02:33.849905 12905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:33.855793 12905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:33.856585 12905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:33.856900 12905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:33.857774 12905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:33.859141 12905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:33.859231 12905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.859336 12905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:33.859402 12905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:33.888665 12905 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:42227
I20260929 03:02:33.888702 13031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:42227 every 8 connection(s)
I20260929 03:02:33.890079 12905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:33.892998 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 12905
I20260929 03:02:33.893194 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:02:33.893752 2870 external_mini_cluster.cc:1428] Setting key f63dcf00545616e58a260906c5685dc5
I20260929 03:02:33.896545 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--builtin_ntp_servers=127.2.205.148:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:02:33.898686 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:33.892592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:54781 (local address 127.2.205.190:45099)
0929 03:02:33.892814 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:33.892821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:33.894543 (+ 1722us) server_negotiation.cc:407] Connection header received
0929 03:02:33.896408 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:33.896413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:33.896482 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:33.896568 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:33.897960 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:33.897967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:33.897972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:33.897993 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:33.898027 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:33.898047 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:33.898149 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:33.898337 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 03:02:33.898412 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:02:33.899765 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:33.893320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45099 (local address 127.2.205.130:54781)
0929 03:02:33.894347 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:33.894396 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:33.896138 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:33.896854 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:33.896863 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:33.896971 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:33.897652 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:33.897655 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:33.897695 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:33.897788 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:33.898131 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:33.898141 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:33.898148 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:02:33.898418 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 03:02:33.898738 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":309,"threads_started":1}
I20260929 03:02:33.901353 13032 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45099
I20260929 03:02:33.901661 13032 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:33.902279 13032 heartbeater.cc:511] Master 127.2.205.190:45099 requested a full tablet report, sending...
I20260929 03:02:33.903702 12715 ts_manager.cc:194] Registered new tserver with Master: 0e340c6acaa94107a6759d35d65f4914 (127.2.205.130:42227)
W20260929 03:02:34.218478 13035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:34.218833 13035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:34.218963 13035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:34.226692 13035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:34.226827 13035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:34.226971 13035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:34.236588 13035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:34.237990 13035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:34.239835 13035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:34.252904 13040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:34.252976 13041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:34.252997 13043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:34.254444 13035 server_base.cc:1034] running on GCE node
I20260929 03:02:34.255079 13035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:34.256031 13035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:34.257225 13035 hybrid_clock.cc:648] HybridClock initialized: now 1790650954257171 us; error 48 us; skew 500 ppm
I20260929 03:02:34.257536 13035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:02:34.259660 13035 webserver.cc:460] Webserver started at http://127.2.205.131:41869/ using document root <none> and password file <none>
I20260929 03:02:34.260353 13035 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:34.260452 13035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:34.260802 13035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:34.263592 13035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "497e574400894c0ea260ca6d55a904eb"
format_stamp: "Formatted at 2026-09-29 03:02:34 on dist-test-slave-530n"
server_key: "521d58e82e0f2fe5a3d5a70ef030d17e"
server_key_iv: "d79adea1092c1ff0964b538e1b47a9d0"
server_key_version: "encryptionkey@0"
I20260929 03:02:34.264340 13035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "497e574400894c0ea260ca6d55a904eb"
format_stamp: "Formatted at 2026-09-29 03:02:34 on dist-test-slave-530n"
server_key: "521d58e82e0f2fe5a3d5a70ef030d17e"
server_key_iv: "d79adea1092c1ff0964b538e1b47a9d0"
server_key_version: "encryptionkey@0"
I20260929 03:02:34.269316 13035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 03:02:34.272564 13049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:34.273802 13035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:34.273943 13035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "497e574400894c0ea260ca6d55a904eb"
format_stamp: "Formatted at 2026-09-29 03:02:34 on dist-test-slave-530n"
server_key: "521d58e82e0f2fe5a3d5a70ef030d17e"
server_key_iv: "d79adea1092c1ff0964b538e1b47a9d0"
server_key_version: "encryptionkey@0"
I20260929 03:02:34.274135 13035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:34.280004 13035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:34.280802 13035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:34.281049 13035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:34.281934 13035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:34.283387 13035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:34.283476 13035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:34.283581 13035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:34.283648 13035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:34.313417 13035 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:32917
I20260929 03:02:34.313465 13161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:32917 every 8 connection(s)
I20260929 03:02:34.314723 13035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:34.318272 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 13035
I20260929 03:02:34.318460 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:02:34.319031 2870 external_mini_cluster.cc:1428] Setting key 783772c2042505cf89ff8d24da1afb54
I20260929 03:02:34.323601 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:34.317189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:44345 (local address 127.2.205.190:45099)
0929 03:02:34.317353 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:34.317357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:34.319101 (+ 1744us) server_negotiation.cc:407] Connection header received
0929 03:02:34.321315 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:34.321320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:34.321388 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:34.321495 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:34.322999 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:34.323004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:34.323007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:34.323025 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:34.323061 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:34.323073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:34.323175 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:34.323326 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:02:34.323374 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:02:34.324666 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:34.317739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45099 (local address 127.2.205.131:44345)
0929 03:02:34.318900 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:34.318948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:34.320992 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:34.321787 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:34.321797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:34.321924 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:34.322625 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:02:34.322628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:34.322680 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:02:34.322808 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:34.323155 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:34.323162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:34.323168 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:02:34.323417 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 03:02:34.323647 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":331,"threads_started":1}
I20260929 03:02:34.326103 13162 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45099
I20260929 03:02:34.326404 13162 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:34.326996 13162 heartbeater.cc:511] Master 127.2.205.190:45099 requested a full tablet report, sending...
I20260929 03:02:34.328267 12715 ts_manager.cc:194] Registered new tserver with Master: 497e574400894c0ea260ca6d55a904eb (127.2.205.131:32917)
I20260929 03:02:34.334559 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:02:34.340206 12903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:02:34.338809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.2.205.190:45099)
0929 03:02:34.339015 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:34.339022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:34.339374 (+ 352us) server_negotiation.cc:407] Connection header received
0929 03:02:34.339535 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:34.339539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:34.339597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:34.339675 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:34.339947 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48454: BlockingRecv error: recv got EOF from 127.0.0.1:48454 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:02:34.341077 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12774
I20260929 03:02:34.344430 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:34.341991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.2.205.129:36441)
0929 03:02:34.342458 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:34.342465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:34.342491 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:02:34.342548 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:34.342557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:34.342771 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:34.342893 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:34.343811 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:34.343819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:34.343832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:34.343847 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:34.343903 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:34.343922 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:34.344035 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:34.344117 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:02:34.344208 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":202,"threads_started":1}
I20260929 03:02:34.345757 12896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:34.426062 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12774
I20260929 03:02:34.439669 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12905
I20260929 03:02:34.442960 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:34.440580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.2.205.130:42227)
0929 03:02:34.441098 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:34.441105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:34.441133 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:02:34.441195 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:34.441202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:34.441418 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:34.441544 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:34.442351 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:34.442357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:34.442370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:34.442383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:34.442441 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:34.442463 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:34.442573 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:34.442643 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:02:34.442730 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260929 03:02:34.444175 13027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:34.523695 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12905
I20260929 03:02:34.536438 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 13035
I20260929 03:02:34.540112 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:34.537393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.2.205.131:32917)
0929 03:02:34.538001 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:34.538008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:34.538037 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:02:34.538114 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:34.538120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:34.538339 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:34.538465 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:34.539456 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:34.539462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:34.539474 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:02:34.539486 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:34.539525 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:02:34.539539 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:34.539668 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:34.539715 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:02:34.539802 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":317,"threads_started":1}
I20260929 03:02:34.541428 13157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:34.622094 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 13035
I20260929 03:02:34.634626 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 12684
I20260929 03:02:34.635808 12745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:34.736658 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 12684
2026-09-29T03:02:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2301 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:02:34 dist-test-slave-530n krb5kdc[13180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:02:34 dist-test-slave-530n krb5kdc[13180](info): set up 2 sockets
Sep 29 03:02:34 dist-test-slave-530n krb5kdc[13180](info): commencing operation
krb5kdc: starting...
W20260929 03:02:35.753860 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:02:35 dist-test-slave-530n krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:02:35.972519 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36559
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:36.299204 13193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:36.299546 13193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:36.299664 13193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:36.307246 13193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:02:36.307384 13193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:36.307447 13193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:36.307533 13193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:02:36.307577 13193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:02:36.317997 13193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:36.319373 13193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:36.320727 13193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:36.332873 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:36.332922 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:36.333055 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:36.333838 13193 server_base.cc:1034] running on GCE node
I20260929 03:02:36.334597 13193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:36.336081 13193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:36.337284 13193 hybrid_clock.cc:648] HybridClock initialized: now 1790650956337231 us; error 44 us; skew 500 ppm
I20260929 03:02:36.337622 13193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:36 dist-test-slave-530n krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:36.340675 13193 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:02:36.342193 13193 webserver.cc:460] Webserver started at http://127.2.205.190:40893/ using document root <none> and password file <none>
I20260929 03:02:36.342895 13193 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:36.342999 13193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:36.343315 13193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:36.346418 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "ca0927e373e94f0286e61f63b00f0490"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "bde39c363801aac1606193a8b70e5022"
server_key_iv: "9165c88a5138104ca676a24005a18581"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.347157 13193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "ca0927e373e94f0286e61f63b00f0490"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "bde39c363801aac1606193a8b70e5022"
server_key_iv: "9165c88a5138104ca676a24005a18581"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.352264 13193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:02:36.355656 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:36.357038 13193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:02:36.357180 13193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "ca0927e373e94f0286e61f63b00f0490"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "bde39c363801aac1606193a8b70e5022"
server_key_iv: "9165c88a5138104ca676a24005a18581"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.357335 13193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:36.368661 13193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:36.372310 13193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:36.372532 13193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:36.390031 13193 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36559
I20260929 03:02:36.390067 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36559 every 8 connection(s)
I20260929 03:02:36.391536 13193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:02:36.394345 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 13193
I20260929 03:02:36.394562 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:02:36.395185 2870 external_mini_cluster.cc:1428] Setting key 97c9b61c122b80eb4a4bb9829d247a08
I20260929 03:02:36.395429 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:36.409626 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: Bootstrap starting.
Sep 29 03:02:36 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:36.412997 13260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:36.414075 13260 log.cc:824] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:36.417318 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: No bootstrap required, opened a new log
I20260929 03:02:36.418525 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:36.397143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.2.205.190:36559)
0929 03:02:36.397881 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:36.397895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:36.397936 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:02:36.399955 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:36.399979 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:36.400402 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:36.400772 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:36.402570 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:36.403868 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:36.405677 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:36.406413 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:36.406570 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:36.411097 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:36.411105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:36.411121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:36.411162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:36.414267 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:36.414965 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:36.414976 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:36.414983 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:36.415169 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:36.415561 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:36.415564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:36.415566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:36.416191 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:36.416524 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:36.416978 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 03:02:36.417455 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":330,"threads_started":1}
I20260929 03:02:36.425441 13260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } }
I20260929 03:02:36.425823 13260 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:36.425927 13260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0927e373e94f0286e61f63b00f0490, State: Initialized, Role: FOLLOWER
I20260929 03:02:36.426505 13260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } }
I20260929 03:02:36.426714 13260 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:02:36.426857 13260 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:02:36.427006 13260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:36.429409 13260 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } }
I20260929 03:02:36.429926 13260 leader_election.cc:302] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0927e373e94f0286e61f63b00f0490; no voters:
I20260929 03:02:36.430527 13260 leader_election.cc:288] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:02:36.430689 13265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:36.431294 13265 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [term 1 LEADER]: Becoming Leader. State: Replica: ca0927e373e94f0286e61f63b00f0490, State: Running, Role: LEADER
I20260929 03:02:36.431947 13265 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } }
I20260929 03:02:36.433284 13260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:02:36.434818 13267 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0927e373e94f0286e61f63b00f0490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } } }
I20260929 03:02:36.434865 13266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0927e373e94f0286e61f63b00f0490. Latest consensus state: current_term: 1 leader_uuid: "ca0927e373e94f0286e61f63b00f0490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0927e373e94f0286e61f63b00f0490" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36559 } } }
I20260929 03:02:36.435249 13267 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:36.436372 13266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490 [sys.catalog]: This master's current role is: LEADER
I20260929 03:02:36.437219 13272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:02:36.442701 13272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:02:36.455335 13272 catalog_manager.cc:1347] Generated new cluster ID: 6c98fca7ffd84901b70d50a9313de12d
I20260929 03:02:36.455489 13272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:02:36.472441 13272 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:02:36.474054 13272 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:02:36.489969 13272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: Generated new TSK 0
I20260929 03:02:36.491032 13272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:02:36.561067 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:36.887722 13286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:36.888031 13286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:36.888127 13286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:36.895622 13286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:36.895748 13286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:36.895864 13286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:02:36.906061 13286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:36.907351 13286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:36.908699 13286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:36.922128 13291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:36.922235 13292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:36.922237 13294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:36.923486 13286 server_base.cc:1034] running on GCE node
I20260929 03:02:36.924199 13286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:36.925349 13286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:36.926575 13286 hybrid_clock.cc:648] HybridClock initialized: now 1790650956926496 us; error 73 us; skew 500 ppm
I20260929 03:02:36.926923 13286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:36 dist-test-slave-530n krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:36.930291 13286 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:02:36.931825 13286 webserver.cc:460] Webserver started at http://127.2.205.129:38127/ using document root <none> and password file <none>
I20260929 03:02:36.932569 13286 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:36.932675 13286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:36.932989 13286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:36.936203 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "2de544e89867439bb62183efdb3df148"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "fea7de4e0b8821e5b5f7ae4504084671"
server_key_iv: "fdff1231384541891c94236bedb1a09d"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.937136 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "2de544e89867439bb62183efdb3df148"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "fea7de4e0b8821e5b5f7ae4504084671"
server_key_iv: "fdff1231384541891c94236bedb1a09d"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.942807 13286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:02:36.946542 13301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:36.948060 13286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:02:36.948204 13286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "2de544e89867439bb62183efdb3df148"
format_stamp: "Formatted at 2026-09-29 03:02:36 on dist-test-slave-530n"
server_key: "fea7de4e0b8821e5b5f7ae4504084671"
server_key_iv: "fdff1231384541891c94236bedb1a09d"
server_key_version: "encryptionkey@0"
I20260929 03:02:36.948374 13286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:36.969861 13286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:36.974125 13286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:36.974351 13286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:36.975232 13286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:36.976583 13286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:36.976657 13286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:36.976774 13286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:36.976830 13286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 03:02:37.007893 13286 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34697
I20260929 03:02:37.007937 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34697 every 8 connection(s)
I20260929 03:02:37.009274 13286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:02:37.013269 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 13286
I20260929 03:02:37.013511 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:02:37.014122 2870 external_mini_cluster.cc:1428] Setting key d48df46421a20bcf9fdd846f2e226c5b
Sep 29 03:02:37 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:37.029181 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:37.011777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:38149 (local address 127.2.205.190:36559)
0929 03:02:37.012234 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:37.012241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:37.013255 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 03:02:37.015176 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:37.015182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:37.015266 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:37.015361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:37.017170 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.017883 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.019437 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.019825 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.019938 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:37.023390 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:37.023396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:37.023399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:37.023435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:37.025963 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:37.026736 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:37.026741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:37.026743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:37.026882 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:37.027334 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:37.027338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:37.027341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:37.027627 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:37.027774 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:37.028535 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 03:02:37.028811 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":200,"threads_started":1}
I20260929 03:02:37.030252 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:37.012355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36559 (local address 127.2.205.129:38149)
0929 03:02:37.013082 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:37.013123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:37.014903 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:37.015633 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:37.015643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:37.016286 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:37.016991 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:37.017014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.018034 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.018037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:37.019287 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:37.019300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.020049 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.020057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:37.020292 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:37.020964 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:37.020993 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:37.023162 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:37.026107 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:37.026117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:37.026146 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:37.026603 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:37.026987 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:37.026992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:37.026996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:37.027213 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:37.027761 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:37.027769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:37.028152 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 03:02:37.028405 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 03:02:37.029005 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":257,"threads_started":1}
I20260929 03:02:37.032011 13414 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36559
I20260929 03:02:37.032388 13414 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:37.033456 13414 heartbeater.cc:511] Master 127.2.205.190:36559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:02:37.036006 13225 ts_manager.cc:194] Registered new tserver with Master: 2de544e89867439bb62183efdb3df148 (127.2.205.129:34697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:02:37.037762 13225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:38149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:02:37.062161 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:37.388195 13420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:37.388481 13420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:37.388613 13420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:37.399461 13420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:37.399600 13420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:37.399803 13420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:02:37.414186 13420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:37.415534 13420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:37.416889 13420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:37.429733 13428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:37.429721 13425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:37.429860 13426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:37.430317 13420 server_base.cc:1034] running on GCE node
I20260929 03:02:37.430959 13420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:37.431934 13420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:37.433125 13420 hybrid_clock.cc:648] HybridClock initialized: now 1790650957433078 us; error 53 us; skew 500 ppm
I20260929 03:02:37.433427 13420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:37 dist-test-slave-530n krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:37.436381 13420 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:02:37.437973 13420 webserver.cc:460] Webserver started at http://127.2.205.130:43071/ using document root <none> and password file <none>
I20260929 03:02:37.438669 13420 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:37.438789 13420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:37.439066 13420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:37.441972 13420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "2f505d2b1ec54f70a7fa50ae0446f23b"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "5b87abd99835001c50069cde9aba2b75"
server_key_iv: "7156f3166c0923b661151db1b739b564"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.442744 13420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "2f505d2b1ec54f70a7fa50ae0446f23b"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "5b87abd99835001c50069cde9aba2b75"
server_key_iv: "7156f3166c0923b661151db1b739b564"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.447683 13420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:02:37.450999 13435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:37.452426 13420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:02:37.452561 13420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "2f505d2b1ec54f70a7fa50ae0446f23b"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "5b87abd99835001c50069cde9aba2b75"
server_key_iv: "7156f3166c0923b661151db1b739b564"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.452731 13420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:37.474712 13420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:37.479228 13420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:37.479480 13420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:37.480479 13420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:37.481989 13420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:37.482070 13420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:37.482164 13420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:37.482208 13420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:37.511874 13420 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:39765
I20260929 03:02:37.511950 13547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:39765 every 8 connection(s)
I20260929 03:02:37.513268 13420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:02:37.523468 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 13420
I20260929 03:02:37.523707 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:02:37.524124 2870 external_mini_cluster.cc:1428] Setting key 71ad81f3b21f2a367a2cb6f4b090015f
Sep 29 03:02:37 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:37.533901 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:37.515770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:50083 (local address 127.2.205.190:36559)
0929 03:02:37.515929 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:37.515933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:37.517151 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 03:02:37.518982 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:37.518987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:37.519069 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:37.519156 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:37.520702 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.521501 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.523121 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.523535 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.523669 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:37.527781 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:37.527787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:37.527790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:37.527824 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:37.530670 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:37.531513 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:37.531519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:37.531522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:37.531675 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:37.532054 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:37.532059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:37.532062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:37.532386 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:37.532524 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:37.533314 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 03:02:37.533566 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:02:37.534992 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:37.516229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36559 (local address 127.2.205.130:50083)
0929 03:02:37.516964 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:37.517003 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:37.518694 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:37.519359 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:37.519367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:37.519876 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:37.520461 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:37.520484 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.521667 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.521671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:37.522949 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:37.522962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:37.523787 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:37.523795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:37.524037 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:37.524978 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:37.525003 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:37.527521 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:37.530841 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:37.530852 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:37.530872 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:37.531396 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:37.531773 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:37.531776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:37.531778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:37.531965 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:37.532489 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:37.532495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:37.532921 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 03:02:37.533175 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 03:02:37.533647 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260929 03:02:37.536487 13548 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36559
I20260929 03:02:37.536860 13548 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:37.537725 13548 heartbeater.cc:511] Master 127.2.205.190:36559 requested a full tablet report, sending...
I20260929 03:02:37.539259 13225 ts_manager.cc:194] Registered new tserver with Master: 2f505d2b1ec54f70a7fa50ae0446f23b (127.2.205.130:39765)
I20260929 03:02:37.540087 13225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:50083
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:02:37.573493 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:02:37.891932 13553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:02:37.892230 13553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:02:37.892376 13553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:02:37.899724 13553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:02:37.899850 13553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:02:37.900005 13553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:02:37.910190 13553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:36559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:02:37.911578 13553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:02:37.913065 13553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:02:37.926025 13559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:37.926064 13561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:02:37.926069 13558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:02:37.927093 13553 server_base.cc:1034] running on GCE node
I20260929 03:02:37.927722 13553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:02:37.928733 13553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:02:37.929968 13553 hybrid_clock.cc:648] HybridClock initialized: now 1790650957929919 us; error 47 us; skew 500 ppm
I20260929 03:02:37.930298 13553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:02:37 dist-test-slave-530n krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:02:37.939074 13553 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:02:37.940626 13553 webserver.cc:460] Webserver started at http://127.2.205.131:37559/ using document root <none> and password file <none>
I20260929 03:02:37.941382 13553 fs_manager.cc:359] Metadata directory not provided
I20260929 03:02:37.941488 13553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:02:37.941746 13553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:02:37.944666 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "7d0c19dc5f0740fbb1d3747acb9e152d"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "7dc81d582ef09a867a3ce9fe935abfaa"
server_key_iv: "0b0c23ce9c655184836bd05ec9aa1084"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.945457 13553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "7d0c19dc5f0740fbb1d3747acb9e152d"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "7dc81d582ef09a867a3ce9fe935abfaa"
server_key_iv: "0b0c23ce9c655184836bd05ec9aa1084"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.950529 13553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:02:37.953883 13568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:37.955175 13553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:02:37.955312 13553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "7d0c19dc5f0740fbb1d3747acb9e152d"
format_stamp: "Formatted at 2026-09-29 03:02:37 on dist-test-slave-530n"
server_key: "7dc81d582ef09a867a3ce9fe935abfaa"
server_key_iv: "0b0c23ce9c655184836bd05ec9aa1084"
server_key_version: "encryptionkey@0"
I20260929 03:02:37.955482 13553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:02:37.969620 13553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:02:37.974246 13553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:02:37.974545 13553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:02:37.975407 13553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:02:37.976718 13553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:02:37.976821 13553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:37.976936 13553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:02:37.976990 13553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:02:38.006565 13553 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:34565
I20260929 03:02:38.006590 13680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:34565 every 8 connection(s)
I20260929 03:02:38.007915 13553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:02:38.014459 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 13553
I20260929 03:02:38.014731 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:02:38.015228 2870 external_mini_cluster.cc:1428] Setting key 57e2377204dab0ac5016c3d4b9709580
Sep 29 03:02:38 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650957, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:02:38.028630 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.010957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:52339 (local address 127.2.205.190:36559)
0929 03:02:38.011178 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.011186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.012556 (+ 1370us) server_negotiation.cc:407] Connection header received
0929 03:02:38.014626 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.014634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.014764 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.014885 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:38.016671 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.017503 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.019086 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.019501 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.019610 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.023098 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:38.023103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:38.023107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:38.023149 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:38.025633 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.026346 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.026350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.026352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.026484 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.026896 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.026900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.026902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.027182 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:38.027325 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.028040 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.028279 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:02:38.029738 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.011448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:36559 (local address 127.2.205.131:52339)
0929 03:02:38.012334 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.012395 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.014283 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.015115 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.015125 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.015723 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:02:38.016497 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.016525 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.017660 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.017663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.018941 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.018953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.019735 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.019743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.019978 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.020662 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:02:38.020687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:02:38.022875 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:02:38.025778 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:38.025788 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:38.025807 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:38.026226 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:38.026586 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:02:38.026591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:02:38.026595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:02:38.026789 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:02:38.027309 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:02:38.027317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:02:38.027689 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.027938 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.028501 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":310,"threads_started":1}
I20260929 03:02:38.031244 13681 heartbeater.cc:347] Connected to a master server at 127.2.205.190:36559
I20260929 03:02:38.031533 13681 heartbeater.cc:464] Registering TS with master...
I20260929 03:02:38.032258 13681 heartbeater.cc:511] Master 127.2.205.190:36559 requested a full tablet report, sending...
I20260929 03:02:38.033682 13225 ts_manager.cc:194] Registered new tserver with Master: 7d0c19dc5f0740fbb1d3747acb9e152d (127.2.205.131:34565)
I20260929 03:02:38.034543 13225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:52339
I20260929 03:02:38.040853 13414 heartbeater.cc:503] Master 127.2.205.190:36559 was elected leader, sending a full tablet report...
I20260929 03:02:38.043711 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:02:38.059695 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.048122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.2.205.190:36559)
0929 03:02:38.048280 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.048284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.048726 (+ 442us) server_negotiation.cc:407] Connection header received
0929 03:02:38.048905 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.048908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.048970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.049039 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:38.050349 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.051063 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.052343 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.052783 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.052874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.054915 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:38.054919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:38.054922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:38.054952 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:38.057391 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.057985 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.057990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.057992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.058122 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.058472 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.058477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.058479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.058787 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:38.058958 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.059259 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.059421 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:02:38.063397 13225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:02:38.069952 13225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:02:38.096725 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.082731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.2.205.131:34565)
0929 03:02:38.083933 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.083943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.085658 (+ 1715us) server_negotiation.cc:407] Connection header received
0929 03:02:38.086000 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.086010 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.086330 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.086520 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:38.087657 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.089490 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.093897 (+ 4407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.095528 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.095665 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.095910 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.096200 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.096377 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":1461,"threads_started":1}
I20260929 03:02:38.097249 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.082104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:34565 (local address 127.0.0.1:45402)
0929 03:02:38.084321 (+ 2217us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.084338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.085792 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.086929 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.086939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.086974 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:38.087422 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.087441 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.090992 (+ 3551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.090998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.093684 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.093703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.095732 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.095739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.095938 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.095986 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.096673 (+ 687us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.096896 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2109,"thread_start_us":898,"threads_started":1}
I20260929 03:02:38.101477 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.081664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34697 (local address 127.0.0.1:35264)
0929 03:02:38.082983 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.083027 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.083212 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.094663 (+ 11451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.094668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.094690 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:38.095069 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.095081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.097041 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.097047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.098926 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.098936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.100719 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.100725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.100907 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.100940 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.101021 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.101147 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":2418,"threads_started":1}
I20260929 03:02:38.101531 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.081755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.2.205.129:34697)
0929 03:02:38.093982 (+ 12227us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.093989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.094016 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:02:38.094126 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.094135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.094424 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.095288 (+ 864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:38.095534 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.096883 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.099120 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.100508 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.100624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.100875 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.101088 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.101247 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12074,"thread_start_us":11917,"threads_started":1}
I20260929 03:02:38.101330 13616 tablet_service.cc:1467] Processing CreateTablet for tablet 8b838c6e931c499085cf8c9b920038b0 (DEFAULT_TABLE table=test-table [id=fdde047f021b4728a307ef2bb8914f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:38.102979 13616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b838c6e931c499085cf8c9b920038b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:38.103618 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.083898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:39765 (local address 127.0.0.1:49506)
0929 03:02:38.093091 (+ 9193us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.093113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.093248 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.096133 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.096137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.096163 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:38.096511 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.096522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.098621 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.098626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.100640 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.100659 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.102892 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.102898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.103060 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.103092 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.103195 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.103323 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9016,"thread_start_us":9441,"threads_started":1}
I20260929 03:02:38.103708 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.084362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.2.205.130:39765)
0929 03:02:38.095566 (+ 11204us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.095575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.095603 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:02:38.095712 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.095720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.095938 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.096131 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:38.096682 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.098479 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.100854 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.102688 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.102825 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.103001 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.103272 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.103430 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11053,"thread_start_us":10964,"threads_started":1}
I20260929 03:02:38.104915 13349 tablet_service.cc:1467] Processing CreateTablet for tablet 8b838c6e931c499085cf8c9b920038b0 (DEFAULT_TABLE table=test-table [id=fdde047f021b4728a307ef2bb8914f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:38.106350 13349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b838c6e931c499085cf8c9b920038b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:38.107232 13483 tablet_service.cc:1467] Processing CreateTablet for tablet 8b838c6e931c499085cf8c9b920038b0 (DEFAULT_TABLE table=test-table [id=fdde047f021b4728a307ef2bb8914f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:02:38.108521 13483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b838c6e931c499085cf8c9b920038b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:02:38.114892 13699 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Bootstrap starting.
I20260929 03:02:38.118202 13699 tablet_bootstrap.cc:654] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:38.119333 13699 log.cc:824] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:38.123051 13701 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Bootstrap starting.
I20260929 03:02:38.123100 13702 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Bootstrap starting.
I20260929 03:02:38.123418 13699 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: No bootstrap required, opened a new log
I20260929 03:02:38.123701 13699 ts_tablet_manager.cc:1397] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 03:02:38.126124 13702 tablet_bootstrap.cc:654] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:38.126533 13701 tablet_bootstrap.cc:654] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Neither blocks nor log segments found. Creating new log.
I20260929 03:02:38.127243 13702 log.cc:824] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:38.127804 13701 log.cc:824] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Log is configured to *not* fsync() on all Append() calls
I20260929 03:02:38.130316 13701 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: No bootstrap required, opened a new log
I20260929 03:02:38.130460 13702 tablet_bootstrap.cc:492] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: No bootstrap required, opened a new log
I20260929 03:02:38.130692 13701 ts_tablet_manager.cc:1397] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 03:02:38.130719 13702 ts_tablet_manager.cc:1397] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 03:02:38.134318 13699 raft_consensus.cc:348] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.134819 13699 raft_consensus.cc:374] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:38.134919 13699 raft_consensus.cc:729] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0c19dc5f0740fbb1d3747acb9e152d, State: Initialized, Role: FOLLOWER
I20260929 03:02:38.135550 13699 consensus_queue.cc:260] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.136628 13681 heartbeater.cc:503] Master 127.2.205.190:36559 was elected leader, sending a full tablet report...
I20260929 03:02:38.137312 13699 ts_tablet_manager.cc:1428] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 03:02:38.139067 13701 raft_consensus.cc:348] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.139382 13701 raft_consensus.cc:374] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:38.139473 13701 raft_consensus.cc:729] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de544e89867439bb62183efdb3df148, State: Initialized, Role: FOLLOWER
I20260929 03:02:38.139981 13701 consensus_queue.cc:260] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.140285 13702 raft_consensus.cc:348] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.140694 13702 raft_consensus.cc:374] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:02:38.140805 13702 raft_consensus.cc:729] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f505d2b1ec54f70a7fa50ae0446f23b, State: Initialized, Role: FOLLOWER
I20260929 03:02:38.141561 13701 ts_tablet_manager.cc:1428] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 03:02:38.141444 13702 consensus_queue.cc:260] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.142493 13548 heartbeater.cc:503] Master 127.2.205.190:36559 was elected leader, sending a full tablet report...
I20260929 03:02:38.144253 13702 ts_tablet_manager.cc:1428] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260929 03:02:38.259999 13682 tablet.cc:2369] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:02:38.261458 13415 tablet.cc:2369] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:02:38.265093 13549 tablet.cc:2369] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:02:38.375509 13707 raft_consensus.cc:482] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:02:38.375814 13707 raft_consensus.cc:504] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.377947 13707 leader_election.cc:288] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2de544e89867439bb62183efdb3df148 (127.2.205.129:34697), 7d0c19dc5f0740fbb1d3747acb9e152d (127.2.205.131:34565)
I20260929 03:02:38.391304 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.378899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:42779 (local address 127.2.205.129:34697)
0929 03:02:38.379090 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.379096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.385449 (+ 6353us) server_negotiation.cc:407] Connection header received
0929 03:02:38.385761 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.385767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.385846 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.385908 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:38.386741 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.387439 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.389494 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.390385 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.390548 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.390641 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.390922 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.391073 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:02:38.391422 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.378406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34697 (local address 127.2.205.130:42779)
0929 03:02:38.385281 (+ 6875us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.385309 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.385582 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.386113 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.386116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.386140 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:38.386579 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.386593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.387581 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.387586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.389330 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.389343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.390609 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.390616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.390768 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.390793 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.390880 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.391021 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6693,"mutex_wait_us":88,"thread_start_us":6785,"threads_started":1}
I20260929 03:02:38.393725 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.379358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:60751 (local address 127.2.205.131:34565)
0929 03:02:38.379532 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.379537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.385683 (+ 6146us) server_negotiation.cc:407] Connection header received
0929 03:02:38.385805 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.385810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.385881 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.385960 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:02:38.386855 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.387586 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.391320 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.392521 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.392657 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.392730 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.393328 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.393476 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:02:38.393718 13369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b838c6e931c499085cf8c9b920038b0" candidate_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de544e89867439bb62183efdb3df148" is_pre_election: true
I20260929 03:02:38.394287 13369 raft_consensus.cc:2454] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f505d2b1ec54f70a7fa50ae0446f23b in term 0.
I20260929 03:02:38.394452 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.378450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:34565 (local address 127.2.205.130:60751)
0929 03:02:38.385526 (+ 7076us) negotiation.cc:107] Waiting for socket to connect
0929 03:02:38.385551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 03:02:38.385700 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:02:38.386109 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:02:38.386114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:02:38.386213 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:02:38.386694 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.386705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.387730 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.387736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.389305 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:02:38.389323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.392937 (+ 3614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.392943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:02:38.393133 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.393183 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:02:38.393906 (+ 723us) client_negotiation.cc:240] Negotiation successful
0929 03:02:38.394057 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6953,"mutex_wait_us":25,"spinlock_wait_cycles":26880,"thread_start_us":6404,"threads_started":1}
I20260929 03:02:38.395012 13438 leader_election.cc:302] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2de544e89867439bb62183efdb3df148, 2f505d2b1ec54f70a7fa50ae0446f23b; no voters:
I20260929 03:02:38.395442 13707 raft_consensus.cc:2781] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:02:38.395618 13707 raft_consensus.cc:482] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:02:38.395473 13636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b838c6e931c499085cf8c9b920038b0" candidate_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" is_pre_election: true
I20260929 03:02:38.395720 13707 raft_consensus.cc:3037] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:38.396029 13636 raft_consensus.cc:2454] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f505d2b1ec54f70a7fa50ae0446f23b in term 0.
I20260929 03:02:38.398371 13707 raft_consensus.cc:504] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.398965 13707 leader_election.cc:288] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [CANDIDATE]: Term 1 election: Requested vote from peers 2de544e89867439bb62183efdb3df148 (127.2.205.129:34697), 7d0c19dc5f0740fbb1d3747acb9e152d (127.2.205.131:34565)
I20260929 03:02:38.399473 13636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b838c6e931c499085cf8c9b920038b0" candidate_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d"
I20260929 03:02:38.399510 13369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b838c6e931c499085cf8c9b920038b0" candidate_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de544e89867439bb62183efdb3df148"
I20260929 03:02:38.399685 13636 raft_consensus.cc:3037] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:38.399703 13369 raft_consensus.cc:3037] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:02:38.402846 13636 raft_consensus.cc:2454] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f505d2b1ec54f70a7fa50ae0446f23b in term 1.
I20260929 03:02:38.402846 13369 raft_consensus.cc:2454] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f505d2b1ec54f70a7fa50ae0446f23b in term 1.
I20260929 03:02:38.403308 13436 leader_election.cc:302] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f505d2b1ec54f70a7fa50ae0446f23b, 7d0c19dc5f0740fbb1d3747acb9e152d; no voters:
I20260929 03:02:38.403604 13707 raft_consensus.cc:2781] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:02:38.404029 13707 raft_consensus.cc:686] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 1 LEADER]: Becoming Leader. State: Replica: 2f505d2b1ec54f70a7fa50ae0446f23b, State: Running, Role: LEADER
I20260929 03:02:38.404573 13707 consensus_queue.cc:237] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } }
I20260929 03:02:38.409128 13224 catalog_manager.cc:5495] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f505d2b1ec54f70a7fa50ae0446f23b (127.2.205.130). New cstate: current_term: 1 leader_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f505d2b1ec54f70a7fa50ae0446f23b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39765 } health_report { overall_health: HEALTHY } } }
I20260929 03:02:38.460707 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.455463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.2.205.130:39765)
0929 03:02:38.455633 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.455637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.455654 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:02:38.455743 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.455747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.455823 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.455947 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:02:38.456248 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.457113 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.458426 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.458908 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.459014 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.459528 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:02:38.460109 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:02:38.460237 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.460341 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.460407 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:02:38.468731 13636 raft_consensus.cc:1261] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 1 FOLLOWER]: Refusing update from remote peer 2f505d2b1ec54f70a7fa50ae0446f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:38.468731 13369 raft_consensus.cc:1261] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 1 FOLLOWER]: Refusing update from remote peer 2f505d2b1ec54f70a7fa50ae0446f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:02:38.469817 13710 consensus_queue.cc:1035] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de544e89867439bb62183efdb3df148" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:38.470139 13707 consensus_queue.cc:1035] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0c19dc5f0740fbb1d3747acb9e152d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:02:38.481094 13713 mvcc.cc:205] Tried to move back new op lower bound from 7334506325878050816 to 7334506325628952576. Current Snapshot: MvccSnapshot[applied={T|T < 7334506325878050816}]
I20260929 03:02:38.494644 13224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:38.494926 13224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:02:38.498402 13224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca0927e373e94f0286e61f63b00f0490: Sending DeleteTablet for 3 replicas of tablet 8b838c6e931c499085cf8c9b920038b0
I20260929 03:02:38.499629 13483 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b838c6e931c499085cf8c9b920038b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:38 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:49506
I20260929 03:02:38.499608 13349 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b838c6e931c499085cf8c9b920038b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:38 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:35264
I20260929 03:02:38.499696 13616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b838c6e931c499085cf8c9b920038b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:02:38 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45402
I20260929 03:02:38.501015 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 13286
I20260929 03:02:38.501060 13722 tablet_replica.cc:330] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: stopping tablet replica
I20260929 03:02:38.501523 13722 raft_consensus.cc:2229] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:02:38.502267 13722 raft_consensus.cc:2258] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:38.503037 13723 tablet_replica.cc:330] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: stopping tablet replica
I20260929 03:02:38.503322 13724 tablet_replica.cc:330] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: stopping tablet replica
I20260929 03:02:38.503497 13723 raft_consensus.cc:2229] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:38.503834 13724 raft_consensus.cc:2229] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:02:38.504045 13723 raft_consensus.cc:2258] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:38.504582 13724 raft_consensus.cc:2258] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:02:38.506458 13722 ts_tablet_manager.cc:1905] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:38.506480 13723 ts_tablet_manager.cc:1905] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:38.513418 13724 ts_tablet_manager.cc:1905] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:02:38.513955 13722 ts_tablet_manager.cc:1918] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:38.514204 13722 log.cc:1196] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/8b838c6e931c499085cf8c9b920038b0
I20260929 03:02:38.514226 13723 ts_tablet_manager.cc:1918] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:38.514400 13723 log.cc:1196] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/8b838c6e931c499085cf8c9b920038b0
I20260929 03:02:38.514918 13722 ts_tablet_manager.cc:1939] T 8b838c6e931c499085cf8c9b920038b0 P 2f505d2b1ec54f70a7fa50ae0446f23b: Deleting consensus metadata
I20260929 03:02:38.514925 13723 ts_tablet_manager.cc:1939] T 8b838c6e931c499085cf8c9b920038b0 P 2de544e89867439bb62183efdb3df148: Deleting consensus metadata
I20260929 03:02:38.516808 13209 catalog_manager.cc:4841] TS 2f505d2b1ec54f70a7fa50ae0446f23b (127.2.205.130:39765): tablet 8b838c6e931c499085cf8c9b920038b0 (table test-table [id=fdde047f021b4728a307ef2bb8914f90]) successfully deleted
I20260929 03:02:38.516788 13211 catalog_manager.cc:4841] TS 2de544e89867439bb62183efdb3df148 (127.2.205.129:34697): tablet 8b838c6e931c499085cf8c9b920038b0 (table test-table [id=fdde047f021b4728a307ef2bb8914f90]) successfully deleted
Sep 29 03:02:38 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:02:38.520823 13724 ts_tablet_manager.cc:1918] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:02:38.520991 13724 log.cc:1196] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/8b838c6e931c499085cf8c9b920038b0
I20260929 03:02:38.521523 13724 ts_tablet_manager.cc:1939] T 8b838c6e931c499085cf8c9b920038b0 P 7d0c19dc5f0740fbb1d3747acb9e152d: Deleting consensus metadata
I20260929 03:02:38.523449 13209 catalog_manager.cc:4841] TS 7d0c19dc5f0740fbb1d3747acb9e152d (127.2.205.131:34565): tablet 8b838c6e931c499085cf8c9b920038b0 (table test-table [id=fdde047f021b4728a307ef2bb8914f90]) successfully deleted
I20260929 03:02:38.527655 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.502552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.2.205.129:34697)
0929 03:02:38.502745 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.502752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.504937 (+ 2185us) server_negotiation.cc:407] Connection header received
0929 03:02:38.505237 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.505242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.505323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.505403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:38.507124 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.508234 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.510069 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.510635 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.510754 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.520661 (+ 9907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:38.520678 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:38.520697 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:38.520744 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:38.524389 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.525348 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.525358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.525362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.525539 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.525990 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.525996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.526000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.526450 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:38.526769 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.527060 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.527284 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:02:38.528363 13409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:38.619675 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 13286
I20260929 03:02:38.641531 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 13420
Sep 29 03:02:38 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:02:38.659197 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.642341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.2.205.130:39765)
0929 03:02:38.642504 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.642508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.642559 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:02:38.642741 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.642745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.642823 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.642904 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:38.644434 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.645310 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.647119 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.647583 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.647702 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.652955 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:38.652971 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:38.652988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:38.653047 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:38.656058 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.656911 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.656919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.656921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.657086 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.657477 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.657481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.657483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.657940 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:38.658273 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.658575 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.658817 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:02:38.659943 13543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:38.771783 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 13420
I20260929 03:02:38.787140 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 13553
Sep 29 03:02:38 dist-test-slave-530n krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:02:38.801240 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:02:38.787901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.2.205.131:34565)
0929 03:02:38.788053 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:02:38.788057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:02:38.788096 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:02:38.788270 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:02:38.788274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:02:38.788346 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:02:38.788428 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:02:38.789712 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.790410 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.791534 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:02:38.791960 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:02:38.792057 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:02:38.796201 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:02:38.796217 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:02:38.796234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:02:38.796272 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:02:38.798921 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.799542 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.799549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.799551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.799717 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:02:38.799986 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:02:38.799989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:02:38.799991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:02:38.800336 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:02:38.800542 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 03:02:38.800800 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:02:38.800979 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:02:38.801913 13676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:38.889489 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 13553
I20260929 03:02:38.902841 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 13193
I20260929 03:02:38.903934 13255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:02:39.013640 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 13193
2026-09-29T03:02:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4277 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:02:39 dist-test-slave-530n krb5kdc[13732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:02:39 dist-test-slave-530n krb5kdc[13732](info): set up 2 sockets
Sep 29 03:02:39 dist-test-slave-530n krb5kdc[13732](info): commencing operation
krb5kdc: starting...
W20260929 03:02:40.017664 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:02:40 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:02:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangeradmin_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangeradmin_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangerlookup_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangerlookup_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
I20260929 03:02:40.328315 2870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskg3SEC6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/postgres -l logfile start
2026-09-29 03:02:42.764 UTC [13762] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:02:42.764 UTC [13762] LOG: listening on IPv4 address "127.2.205.148", port 45669
2026-09-29 03:02:42.767 UTC [13762] LOG: listening on Unix socket "/tmp/.s.PGSQL.45669"
2026-09-29 03:02:42.787 UTC [13765] LOG: database system was shut down at 2026-09-29 03:02:41 UTC
2026-09-29 03:02:42.791 UTC [13762] LOG: database system is ready to accept connections
I20260929 03:02:43.673264 2870 mini_postgres.cc:96] Postgres bound to 45669
2026-09-29 03:02:43.690 UTC [13773] FATAL: database "slave" does not exist
127.2.205.148:45669 - accepting connections
I20260929 03:02:43.691004 2870 mini_ranger.cc:161] Starting Ranger...
I20260929 03:02:43.717377 2870 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:02:44.148593 2870 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:02:44.148707 2870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin
2026-09-29 03:02:44,407 [I] DB FLAVOR :POSTGRES
2026-09-29 03:02:44,407 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:02:44,408 [I] Checking connection..
2026-09-29 03:02:44,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:02:44,666 [I] Checking connection passed.
2026-09-29 03:02:44,667 [I] --------- Verifying version history table ---------
2026-09-29 03:02:44,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:02:44,929 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:02:44,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:02:45,188 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:02:45,188 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:02:45,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:02:45.438 UTC [13837] WARNING: there is no transaction in progress
2026-09-29 03:02:45,444 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:02:45,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:02:45,703 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:02:45,704 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:02:45,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:02:45,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:02:46,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-530n.c.gcp-upstream.internal','N') ;"
2026-09-29 03:02:46,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:02:46,673 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:02:46,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:02:46,932 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:02:46,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:02:47,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:02:47,430 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:02:47,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:02:48.391 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.406 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.421 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.437 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.458 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.479 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.539 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.546 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.561 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.572 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.583 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.597 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.608 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.614 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.621 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:48.629 UTC [13945] WARNING: there is no transaction in progress
2026-09-29 03:02:49,280 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:02:49,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:02:49,520 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:02:49,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:02:49,779 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:02:49,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:02:50,040 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:02:50,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:02:50,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:02:50,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-530n.c.gcp-upstream.internal';"
2026-09-29 03:02:50,732 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:02:50,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:02:50,964 [I] DB_PATCHES have already been applied
I20260929 03:02:50.970105 2870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskg3SEC6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:02:50.970206 2870 mini_ranger.cc:193] Using host: 127.2.205.148
I20260929 03:02:50.984575 2870 mini_ranger.cc:227] Ranger admin URL: http://127.2.205.148:46105
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:02:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:02:52.011381 2870 mini_ranger.cc:160] Time spent starting Ranger: real 8.320s user 0.008s sys 0.033s
Sep 29, 2026 3:03:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:03:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:03:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:03:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:03:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:03:12.671875 2870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangerkms_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/rangerkms_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.205.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.205.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:03:12 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650992, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:03:12.828635 2870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:03:12.846913 2870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:03:13.638384 2870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:03:13.638542 2870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms
I20260929 03:03:13.638659 2870 mini_ranger_kms.cc:227] Using postgres at 127.2.205.148:45669
2026-09-29 03:03:14,018 [I] DB FLAVOR :POSTGRES
2026-09-29 03:03:14,018 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:03:14,019 [I] Checking connection
2026-09-29 03:03:14,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:03:14,245 [I] connection success
2026-09-29 03:03:14,245 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:03:14,245 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:03:14,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:03:14,464 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:03:14,464 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:03:14,465 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:03:14,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:45669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:03:14,724 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:03:15.158730 2870 mini_ranger_kms.cc:310] Created kms service
I20260929 03:03:15.372390 2870 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:03:15.418056 2870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:03:15 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650992, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:03:15.846889 2870 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:03:15.847208 2870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:03:15.847257 2870 mini_ranger_kms.cc:240] Using host: 127.2.205.148
I20260929 03:03:15.867583 2870 mini_ranger_kms.cc:276] Ranger KMS PID: 14125
I20260929 03:03:15.867762 2870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.205.148:46637
Sep 29, 2026 3:03:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:03:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:03:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:03:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:03:17.984361 2870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.156s user 0.105s sys 0.295s
I20260929 03:03:17.984588 2870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:03:17.984793 2870 mini_ranger_kms.cc:401] 127.2.205.148:46637/kms/v1/keys
Sep 29 03:03:18 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:03:18 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
Sep 29 03:03:18 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650998, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
Sep 29 03:03:21 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:03:21.263993 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:43059
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:43059
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:46637/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:03:21.600728 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:03:21.601035 14176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:03:21.601120 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:03:21.608588 14176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:03:21.608702 14176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:03:21.608771 14176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:03:21.608844 14176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:03:21.608896 14176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:03:21.618927 14176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.205.148:46637/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:43059
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:03:21.620270 14176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:03:21.621902 14176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:03:21.635088 14182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:21.635179 14184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:21.635139 14181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:03:21.635401 14176 server_base.cc:1034] running on GCE node
I20260929 03:03:21.636327 14176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:03:21.637770 14176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:03:21.638973 14176 hybrid_clock.cc:648] HybridClock initialized: now 1790651001638914 us; error 45 us; skew 500 ppm
I20260929 03:03:21.639297 14176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:03:21 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:03:21.646121 14176 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:03:21.647552 14176 webserver.cc:460] Webserver started at http://127.2.205.190:32977/ using document root <none> and password file <none>
I20260929 03:03:21.648159 14176 fs_manager.cc:359] Metadata directory not provided
I20260929 03:03:21.648257 14176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:03:21.648526 14176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:03:21 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:03:21.709270 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "ad6f0cafd77e45eab222217c1f19a0d2"
format_stamp: "Formatted at 2026-09-29 03:03:21 on dist-test-slave-530n"
server_key: "78b2b092f78d8eaecd5d687a694a3493"
server_key_iv: "c49a820d22e978df6652a9d8363afa95"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:21.710395 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "ad6f0cafd77e45eab222217c1f19a0d2"
format_stamp: "Formatted at 2026-09-29 03:03:21 on dist-test-slave-530n"
server_key: "78b2b092f78d8eaecd5d687a694a3493"
server_key_iv: "c49a820d22e978df6652a9d8363afa95"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:21.717075 14176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 03:03:21.738090 14191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:21.739729 14176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 03:03:21.739909 14176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "ad6f0cafd77e45eab222217c1f19a0d2"
format_stamp: "Formatted at 2026-09-29 03:03:21 on dist-test-slave-530n"
server_key: "78b2b092f78d8eaecd5d687a694a3493"
server_key_iv: "c49a820d22e978df6652a9d8363afa95"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:21.740119 14176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:03:21.761359 14176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:03:21.765092 14176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:03:21.765364 14176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:03:21.784152 14176 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:43059
I20260929 03:03:21.784185 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:43059 every 8 connection(s)
I20260929 03:03:21.785593 14176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:03:21.789301 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 14176
I20260929 03:03:21.789494 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:03:21.789517 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:03:21.790132 2870 external_mini_cluster.cc:1428] Setting key 52989ab8dda7a484e777425043601eb9
I20260929 03:03:21.804720 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: Bootstrap starting.
Sep 29 03:03:21 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:03:21.808866 14243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: Neither blocks nor log segments found. Creating new log.
I20260929 03:03:21.810232 14243 log.cc:824] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:03:21.814195 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: No bootstrap required, opened a new log
I20260929 03:03:21.814831 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:21.792083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.2.205.190:43059)
0929 03:03:21.792972 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:21.792988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:21.793039 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:03:21.795109 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:21.795151 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:21.795781 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:21.796309 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:21.797586 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:21.799577 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:21.800963 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:21.802118 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:21.802329 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:21.806140 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:21.806152 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:21.806179 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:21.806241 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:21.809877 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:21.810728 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:21.810743 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:21.810754 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:21.811326 (+ 572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:21.811833 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:21.811839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:21.811846 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:21.812616 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:21.812974 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:21.813383 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 03:03:21.813747 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":355,"threads_started":1}
I20260929 03:03:21.821803 14243 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } }
I20260929 03:03:21.822273 14243 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:03:21.822387 14243 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6f0cafd77e45eab222217c1f19a0d2, State: Initialized, Role: FOLLOWER
I20260929 03:03:21.823037 14243 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } }
I20260929 03:03:21.823293 14243 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:03:21.823426 14243 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:03:21.823572 14243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:03:21.825786 14243 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } }
I20260929 03:03:21.826380 14243 leader_election.cc:302] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad6f0cafd77e45eab222217c1f19a0d2; no voters:
I20260929 03:03:21.826936 14243 leader_election.cc:288] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:03:21.827531 14248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:03:21.828593 14248 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [term 1 LEADER]: Becoming Leader. State: Replica: ad6f0cafd77e45eab222217c1f19a0d2, State: Running, Role: LEADER
I20260929 03:03:21.829166 14248 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } }
I20260929 03:03:21.829537 14243 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:03:21.838604 14250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad6f0cafd77e45eab222217c1f19a0d2. Latest consensus state: current_term: 1 leader_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } } }
I20260929 03:03:21.838728 14249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad6f0cafd77e45eab222217c1f19a0d2" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 43059 } } }
I20260929 03:03:21.839048 14249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:03:21.839046 14250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:03:21.839452 14243 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:03:21.840798 14262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:03:21.852599 14262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:03:21.862646 14262 catalog_manager.cc:1347] Generated new cluster ID: d302606328044d11b7a57270c0aa0442
I20260929 03:03:21.862782 14262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:03:21.876112 14262 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:03:21.877912 14262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:03:21.890122 14262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: Generated new TSK 0
I20260929 03:03:21.891242 14262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:03:22.644438 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:46637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:03:22 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651002, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:03:23.082222 14288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:03:23.082526 14288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:03:23.082705 14288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:03:23.090173 14288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:03:23.090312 14288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:03:23.090514 14288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:03:23.103410 14288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.205.148:46637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:03:23.104848 14288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:03:23.106169 14288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:03:23.122202 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:23.122790 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:23.147425 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:03:23.147545 14288 server_base.cc:1034] running on GCE node
I20260929 03:03:23.148483 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:03:23.149950 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:03:23.151270 14288 hybrid_clock.cc:648] HybridClock initialized: now 1790651003151193 us; error 62 us; skew 500 ppm
I20260929 03:03:23.151680 14288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:03:23 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:03:23.156297 14288 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:03:23.158368 14288 webserver.cc:460] Webserver started at http://127.2.205.129:42215/ using document root <none> and password file <none>
I20260929 03:03:23.159361 14288 fs_manager.cc:359] Metadata directory not provided
I20260929 03:03:23.159498 14288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:03:23.159897 14288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:03:23 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:03:23.181727 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "493a97e82ba943b9828f790f5d593d71"
format_stamp: "Formatted at 2026-09-29 03:03:23 on dist-test-slave-530n"
server_key: "b3c01ac698442c5680b245c2a78bf695"
server_key_iv: "ba61cebe63884150b03bf99bcfceb4a6"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:23.183173 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "493a97e82ba943b9828f790f5d593d71"
format_stamp: "Formatted at 2026-09-29 03:03:23 on dist-test-slave-530n"
server_key: "b3c01ac698442c5680b245c2a78bf695"
server_key_iv: "ba61cebe63884150b03bf99bcfceb4a6"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:23.206350 14288 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.007s sys 0.012s
I20260929 03:03:23.237316 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:23.241700 14288 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.000s
I20260929 03:03:23.241916 14288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "493a97e82ba943b9828f790f5d593d71"
format_stamp: "Formatted at 2026-09-29 03:03:23 on dist-test-slave-530n"
server_key: "b3c01ac698442c5680b245c2a78bf695"
server_key_iv: "ba61cebe63884150b03bf99bcfceb4a6"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:23.242158 14288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:03:23.285053 14288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:03:23.291316 14288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:03:23.291744 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:03:23.299569 14288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:03:23.301695 14288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:03:23.301879 14288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 03:03:23.302096 14288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:03:23.302233 14288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:23.385262 14288 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:44459
I20260929 03:03:23.385316 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:44459 every 8 connection(s)
I20260929 03:03:23.387228 14288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:03:23.396168 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 14288
I20260929 03:03:23.396447 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:03:23.397333 2870 external_mini_cluster.cc:1428] Setting key 99ea30ecb26e067caa986fe88da1dcbf
Sep 29 03:03:23 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:03:23.425894 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:23.393599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:38361 (local address 127.2.205.190:43059)
0929 03:03:23.405986 (+ 12387us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:23.405992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:23.406013 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:03:23.406078 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:23.406083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:23.406184 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:23.406298 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:23.408655 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:23.409861 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:23.411786 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:23.412384 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:23.412510 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:23.417219 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:23.417228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:23.417233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:23.417282 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:23.421468 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:23.422729 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:23.422737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:23.422741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:23.422914 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:23.423546 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:23.423551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:23.423555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:23.423933 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:23.424060 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:23.425239 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0929 03:03:23.425489 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12239,"thread_start_us":12103,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:03:23.428360 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:23.394415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:43059 (local address 127.2.205.129:38361)
0929 03:03:23.402111 (+ 7696us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:23.402177 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:23.405161 (+ 2984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:23.406855 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:23.406869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:23.407502 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:03:23.408396 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:23.408426 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:23.410130 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:23.410135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:23.411585 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:23.411608 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:23.412877 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:23.412885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:23.413199 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:23.414051 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:03:23.414081 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:03:23.416870 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:03:23.421723 (+ 4853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:23.421734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:23.421758 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:23.422526 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:23.423092 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:23.423097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:23.423101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:23.423406 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:23.424182 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:03:23.424191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:03:23.424725 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 03:03:23.426501 (+ 1776us) client_negotiation.cc:240] Negotiation successful
0929 03:03:23.427140 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7321,"thread_start_us":7065,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:03:23.430606 14421 heartbeater.cc:347] Connected to a master server at 127.2.205.190:43059
I20260929 03:03:23.430980 14421 heartbeater.cc:464] Registering TS with master...
I20260929 03:03:23.431963 14421 heartbeater.cc:511] Master 127.2.205.190:43059 requested a full tablet report, sending...
I20260929 03:03:23.434643 14208 ts_manager.cc:194] Registered new tserver with Master: 493a97e82ba943b9828f790f5d593d71 (127.2.205.129:44459)
I20260929 03:03:23.438400 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:38361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:03:23.467724 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:46637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:03:23 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651002, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
Sep 29 03:03:23 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651002, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
W20260929 03:03:23.993438 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:03:23.993971 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:03:23.994160 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:03:24.006107 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:03:24.006363 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:03:24.006578 14427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:03:24.023770 14427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.205.148:46637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:03:24.025923 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:03:24.027958 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:03:24.068277 14436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:24.068902 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:03:24.075675 14427 server_base.cc:1034] running on GCE node
W20260929 03:03:24.076015 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:03:24.076979 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:03:24.078435 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:03:24.079777 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790651004079703 us; error 66 us; skew 500 ppm
I20260929 03:03:24.080209 14427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:03:24.084189 14427 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:03:24.085997 14427 webserver.cc:460] Webserver started at http://127.2.205.130:34227/ using document root <none> and password file <none>
I20260929 03:03:24.086716 14427 fs_manager.cc:359] Metadata directory not provided
I20260929 03:03:24.086824 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:03:24.087141 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:03:24.102475 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "a61d1f9e37f64b78b642b365009d407b"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "3c1b130dc69451695779a9ce096ec208"
server_key_iv: "b1badbe91371e21a7898fe342c56015f"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.103343 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "a61d1f9e37f64b78b642b365009d407b"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "3c1b130dc69451695779a9ce096ec208"
server_key_iv: "b1badbe91371e21a7898fe342c56015f"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.108626 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:03:24.124959 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.126578 14427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:03:24.126758 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "a61d1f9e37f64b78b642b365009d407b"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "3c1b130dc69451695779a9ce096ec208"
server_key_iv: "b1badbe91371e21a7898fe342c56015f"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.126978 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:03:24.149942 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:03:24.154552 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:03:24.154829 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:03:24.155787 14427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:03:24.157182 14427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:03:24.157258 14427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.157352 14427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:03:24.157402 14427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.193809 14427 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:37915
I20260929 03:03:24.193836 14563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:37915 every 8 connection(s)
I20260929 03:03:24.195114 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:03:24.204233 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 14427
I20260929 03:03:24.204396 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:03:24.204974 2870 external_mini_cluster.cc:1428] Setting key 16313927ecbe7b437d5383e42344e822
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:03:24.217343 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.197801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:58529 (local address 127.2.205.190:43059)
0929 03:03:24.198343 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.198349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.199792 (+ 1443us) server_negotiation.cc:407] Connection header received
0929 03:03:24.202413 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.202417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.202483 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.202571 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:24.204633 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.205692 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.207680 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.208105 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.208217 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.211477 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:24.211482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:24.211486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:24.211519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:24.214116 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.214893 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.214900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.214902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.215036 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.215484 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.215487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.215489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.215779 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:24.215918 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.216642 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.216947 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260929 03:03:24.218672 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.198369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:43059 (local address 127.2.205.130:58529)
0929 03:03:24.199591 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:24.199655 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:24.202076 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:24.202886 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:24.202899 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:24.203526 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:03:24.204410 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.204437 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.205864 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.205868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.207534 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.207554 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.208389 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.208397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.208611 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.209274 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:03:24.209296 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:03:24.211248 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:03:24.214303 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:24.214313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:24.214328 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:24.214771 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:24.215145 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:24.215150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:24.215154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:24.215375 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:24.215894 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:03:24.215903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:03:24.216280 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 03:03:24.216537 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 03:03:24.217279 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":702,"threads_started":1}
I20260929 03:03:24.220311 14564 heartbeater.cc:347] Connected to a master server at 127.2.205.190:43059
I20260929 03:03:24.220613 14564 heartbeater.cc:464] Registering TS with master...
I20260929 03:03:24.221680 14564 heartbeater.cc:511] Master 127.2.205.190:43059 requested a full tablet report, sending...
I20260929 03:03:24.223492 14208 ts_manager.cc:194] Registered new tserver with Master: a61d1f9e37f64b78b642b365009d407b (127.2.205.130:37915)
I20260929 03:03:24.224323 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:58529
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:03:24.264633 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:46637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:03:24.443037 14421 heartbeater.cc:503] Master 127.2.205.190:43059 was elected leader, sending a full tablet report...
W20260929 03:03:24.632184 14570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:03:24.632508 14570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:03:24.632622 14570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:03:24.639956 14570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:03:24.640079 14570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:03:24.640192 14570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:03:24.650503 14570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.205.148:46637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:03:24.651739 14570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:03:24.653400 14570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:03:24.665771 14575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:24.665848 14578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:03:24.665855 14576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:03:24.666440 14570 server_base.cc:1034] running on GCE node
I20260929 03:03:24.666987 14570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:03:24.667869 14570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:03:24.669080 14570 hybrid_clock.cc:648] HybridClock initialized: now 1790651004669023 us; error 42 us; skew 500 ppm
I20260929 03:03:24.669381 14570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:03:24.672247 14570 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:03:24.673628 14570 webserver.cc:460] Webserver started at http://127.2.205.131:36705/ using document root <none> and password file <none>
I20260929 03:03:24.674255 14570 fs_manager.cc:359] Metadata directory not provided
I20260929 03:03:24.674371 14570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:03:24.674692 14570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:03:24.690441 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "61e33cc2a3b04d59b3deabb7f908a7c9"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "85a911bb7d8183ce28fdab0dde9d252c"
server_key_iv: "956062ffc5eb113170560e2c38b5afef"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.691357 14570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "61e33cc2a3b04d59b3deabb7f908a7c9"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "85a911bb7d8183ce28fdab0dde9d252c"
server_key_iv: "956062ffc5eb113170560e2c38b5afef"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.696971 14570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:03:24.713737 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.715817 14570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260929 03:03:24.716009 14570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "61e33cc2a3b04d59b3deabb7f908a7c9"
format_stamp: "Formatted at 2026-09-29 03:03:24 on dist-test-slave-530n"
server_key: "85a911bb7d8183ce28fdab0dde9d252c"
server_key_iv: "956062ffc5eb113170560e2c38b5afef"
server_key_version: "kuduclusterkey@0"
I20260929 03:03:24.716250 14570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:03:24.745208 14570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:03:24.749027 14570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:03:24.749295 14570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:03:24.750226 14570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:03:24.751566 14570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:03:24.751649 14570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.751762 14570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:03:24.751817 14570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:03:24.783088 14570 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:42255
I20260929 03:03:24.783123 14697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:42255 every 8 connection(s)
I20260929 03:03:24.784415 14570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:03:24.789791 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 14570
I20260929 03:03:24.789960 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:03:24.790575 2870 external_mini_cluster.cc:1428] Setting key af833b9157aba9e402d78127f4b70f06
Sep 29 03:03:24 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:03:24.803793 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.787212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:54909 (local address 127.2.205.190:43059)
0929 03:03:24.787699 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.787705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.788831 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 03:03:24.791036 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.791042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.791135 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.791239 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:24.793051 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.793894 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.795362 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.795763 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.795849 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.798882 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:24.798887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:24.798890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:24.798922 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:24.801280 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.801909 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.801913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.801915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.802025 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.802380 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.802382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.802385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.802646 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:24.802750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.803385 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.803543 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":221,"threads_started":1}
I20260929 03:03:24.804931 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.787857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:43059 (local address 127.2.205.131:54909)
0929 03:03:24.788649 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:24.788702 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:24.790732 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:24.791567 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:24.791579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:24.792132 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:03:24.792899 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.792920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.794020 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.794024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.795253 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.795265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.795921 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.795925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.796123 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.796717 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:03:24.796738 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:03:24.798684 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:03:24.801398 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:24.801404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:24.801415 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:24.801824 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:24.802110 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:03:24.802114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:03:24.802118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:03:24.802300 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:03:24.802768 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:03:24.802775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:03:24.803078 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 03:03:24.803304 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 03:03:24.803725 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260929 03:03:24.806170 14698 heartbeater.cc:347] Connected to a master server at 127.2.205.190:43059
I20260929 03:03:24.806433 14698 heartbeater.cc:464] Registering TS with master...
I20260929 03:03:24.807112 14698 heartbeater.cc:511] Master 127.2.205.190:43059 requested a full tablet report, sending...
I20260929 03:03:24.808501 14208 ts_manager.cc:194] Registered new tserver with Master: 61e33cc2a3b04d59b3deabb7f908a7c9 (127.2.205.131:42255)
I20260929 03:03:24.809369 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:54909
I20260929 03:03:24.818023 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:03:24.834861 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.823375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.2.205.190:43059)
0929 03:03:24.823536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.823540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.823596 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:03:24.823760 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.823763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.823858 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.823933 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:24.825386 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.826105 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.827326 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.827709 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.827803 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.830083 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:24.830088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:24.830091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:24.830122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:24.832701 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.833319 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.833323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.833325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.833456 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:24.833750 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:24.833752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:24.833754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:24.834012 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:24.834207 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.834414 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.834574 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:03:24.838649 14208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:03:24.846866 14208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:03:24.877595 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.860064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44459 (local address 127.0.0.1:55944)
0929 03:03:24.868084 (+ 8020us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:24.868107 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:24.868256 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:24.869449 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:24.869459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:24.869494 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:03:24.870082 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.870101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.872384 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.872391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.874702 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.874718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.876421 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.876426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.876657 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.876724 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 03:03:24.876925 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 03:03:24.877155 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7869,"mutex_wait_us":50,"thread_start_us":7738,"threads_started":1}
I20260929 03:03:24.877653 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.860805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.2.205.129:44459)
0929 03:03:24.866190 (+ 5385us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.866201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.868411 (+ 2210us) server_negotiation.cc:407] Connection header received
0929 03:03:24.868556 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.868567 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.868922 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.869139 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:03:24.870360 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.872190 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.874908 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.876168 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.876302 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.876581 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.877094 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.877279 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5141,"thread_start_us":4920,"threads_started":1}
I20260929 03:03:24.879388 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.859498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:37915 (local address 127.0.0.1:42376)
0929 03:03:24.866661 (+ 7163us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:24.866714 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:24.866940 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:24.869685 (+ 2745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:24.869694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:24.869725 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:03:24.870294 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.870307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.873435 (+ 3128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.873442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.875799 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.875820 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.878501 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.878509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.878705 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.878741 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:03:24.878799 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:03:24.878963 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6897,"mutex_wait_us":5,"thread_start_us":6733,"threads_started":1}
I20260929 03:03:24.880522 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.859735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.2.205.130:37915)
0929 03:03:24.868986 (+ 9251us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.868996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.869021 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:03:24.869112 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.869119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.869419 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.870510 (+ 1091us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:03:24.870751 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.873253 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.876061 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.878253 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.879644 (+ 1391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.879880 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.879961 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.880171 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9104,"thread_start_us":8961,"threads_started":1}
I20260929 03:03:24.882637 14356 tablet_service.cc:1467] Processing CreateTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 (DEFAULT_TABLE table=test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:03:24.884361 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31397c9b3f54ce9a53cefc43f1871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:03:24.884321 14499 tablet_service.cc:1467] Processing CreateTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 (DEFAULT_TABLE table=test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:03:24.885680 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.867680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.2.205.131:42255)
0929 03:03:24.874010 (+ 6330us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:24.874018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:24.877320 (+ 3302us) server_negotiation.cc:407] Connection header received
0929 03:03:24.877462 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:24.877471 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:24.877931 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:24.878177 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:03:24.879029 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.880543 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.882546 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.883972 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.884104 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.885109 (+ 1005us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:24.885183 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:03:24.885350 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6144,"thread_start_us":5967,"threads_started":1}
I20260929 03:03:24.885844 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31397c9b3f54ce9a53cefc43f1871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:03:24.886101 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:24.869082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:42255 (local address 127.0.0.1:60510)
0929 03:03:24.877050 (+ 7968us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:24.877068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:24.877183 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:24.878180 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:24.878185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:24.878217 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:03:24.878651 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.878664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.880737 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.880742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.882341 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:24.882360 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:24.884206 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:24.884212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:24.884386 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:24.884421 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:03:24.884598 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 03:03:24.884784 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7844,"thread_start_us":7693,"threads_started":1}
I20260929 03:03:24.889982 14633 tablet_service.cc:1467] Processing CreateTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 (DEFAULT_TABLE table=test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:03:24.891773 14633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31397c9b3f54ce9a53cefc43f1871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:03:24.896318 14717 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Bootstrap starting.
I20260929 03:03:24.900350 14718 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Bootstrap starting.
I20260929 03:03:24.900481 14717 tablet_bootstrap.cc:654] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Neither blocks nor log segments found. Creating new log.
I20260929 03:03:24.901860 14717 log.cc:824] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Log is configured to *not* fsync() on all Append() calls
I20260929 03:03:24.903779 14718 tablet_bootstrap.cc:654] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Neither blocks nor log segments found. Creating new log.
I20260929 03:03:24.905125 14718 log.cc:824] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:03:24.905344 14717 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: No bootstrap required, opened a new log
I20260929 03:03:24.905843 14717 ts_tablet_manager.cc:1397] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260929 03:03:24.906371 14720 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Bootstrap starting.
I20260929 03:03:24.910022 14720 tablet_bootstrap.cc:654] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Neither blocks nor log segments found. Creating new log.
I20260929 03:03:24.911201 14720 log.cc:824] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:03:24.914568 14720 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: No bootstrap required, opened a new log
I20260929 03:03:24.914568 14718 tablet_bootstrap.cc:492] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: No bootstrap required, opened a new log
I20260929 03:03:24.914917 14720 ts_tablet_manager.cc:1397] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 03:03:24.914917 14718 ts_tablet_manager.cc:1397] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260929 03:03:24.916649 14717 raft_consensus.cc:348] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.917092 14717 raft_consensus.cc:374] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:03:24.917218 14717 raft_consensus.cc:729] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493a97e82ba943b9828f790f5d593d71, State: Initialized, Role: FOLLOWER
I20260929 03:03:24.917841 14717 consensus_queue.cc:260] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.922842 14717 ts_tablet_manager.cc:1428] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Time spent starting tablet: real 0.017s user 0.008s sys 0.005s
I20260929 03:03:24.925896 14720 raft_consensus.cc:348] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.926429 14720 raft_consensus.cc:374] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:03:24.926215 14718 raft_consensus.cc:348] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.926537 14720 raft_consensus.cc:729] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e33cc2a3b04d59b3deabb7f908a7c9, State: Initialized, Role: FOLLOWER
I20260929 03:03:24.926645 14718 raft_consensus.cc:374] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:03:24.926734 14718 raft_consensus.cc:729] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61d1f9e37f64b78b642b365009d407b, State: Initialized, Role: FOLLOWER
I20260929 03:03:24.927212 14720 consensus_queue.cc:260] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.927248 14718 consensus_queue.cc:260] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:24.928318 14698 heartbeater.cc:503] Master 127.2.205.190:43059 was elected leader, sending a full tablet report...
I20260929 03:03:24.928767 14564 heartbeater.cc:503] Master 127.2.205.190:43059 was elected leader, sending a full tablet report...
I20260929 03:03:24.929117 14720 ts_tablet_manager.cc:1428] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260929 03:03:24.929404 14718 ts_tablet_manager.cc:1428] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
W20260929 03:03:24.947178 14565 tablet.cc:2369] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:03:25.036010 14699 tablet.cc:2369] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:03:25.089213 14724 raft_consensus.cc:482] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:03:25.089530 14724 raft_consensus.cc:504] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:25.091894 14724 leader_election.cc:288] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61e33cc2a3b04d59b3deabb7f908a7c9 (127.2.205.131:42255), 493a97e82ba943b9828f790f5d593d71 (127.2.205.129:44459)
I20260929 03:03:25.098479 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.092420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:46341 (local address 127.2.205.129:44459)
0929 03:03:25.092802 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.092808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.093187 (+ 379us) server_negotiation.cc:407] Connection header received
0929 03:03:25.093385 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.093388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.093448 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.093521 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:03:25.094183 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.094899 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.096703 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.097579 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.097736 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.097814 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.098110 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.098238 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260929 03:03:25.098495 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.092356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:52857 (local address 127.2.205.131:42255)
0929 03:03:25.092573 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.092580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.093371 (+ 791us) server_negotiation.cc:407] Connection header received
0929 03:03:25.093519 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.093523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.093588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.093666 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:03:25.094266 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.094998 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.096700 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.097635 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.097788 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.097860 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.098153 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.098272 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:03:25.098645 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.092339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:42255 (local address 127.2.205.130:52857)
0929 03:03:25.093257 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:25.093271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:25.093397 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:25.093696 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:25.093700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:25.093742 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:03:25.094120 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:25.094129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.095130 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.095134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:25.096550 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:25.096562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.097855 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.097862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:25.098009 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.098031 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:03:25.098106 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:03:25.098241 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":4,"thread_start_us":621,"threads_started":1}
I20260929 03:03:25.098922 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.092264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:44459 (local address 127.2.205.130:46341)
0929 03:03:25.093057 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 03:03:25.093077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:03:25.093245 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:03:25.093558 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:03:25.093562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:03:25.093586 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:03:25.094024 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:25.094038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.095028 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.095033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:25.096538 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:03:25.096551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.097821 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.097827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:03:25.097961 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.097985 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:03:25.098067 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:03:25.098201 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":21,"spinlock_wait_cycles":21248,"thread_start_us":411,"threads_started":1}
I20260929 03:03:25.099740 14653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31397c9b3f54ce9a53cefc43f1871b0" candidate_uuid: "a61d1f9e37f64b78b642b365009d407b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" is_pre_election: true
I20260929 03:03:25.099828 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31397c9b3f54ce9a53cefc43f1871b0" candidate_uuid: "a61d1f9e37f64b78b642b365009d407b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493a97e82ba943b9828f790f5d593d71" is_pre_election: true
I20260929 03:03:25.100237 14653 raft_consensus.cc:2454] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61d1f9e37f64b78b642b365009d407b in term 0.
I20260929 03:03:25.100294 14376 raft_consensus.cc:2454] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61d1f9e37f64b78b642b365009d407b in term 0.
I20260929 03:03:25.100955 14453 leader_election.cc:302] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 493a97e82ba943b9828f790f5d593d71, 61e33cc2a3b04d59b3deabb7f908a7c9, a61d1f9e37f64b78b642b365009d407b; no voters:
I20260929 03:03:25.101800 14724 raft_consensus.cc:2781] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:03:25.101964 14724 raft_consensus.cc:482] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:03:25.102062 14724 raft_consensus.cc:3037] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:03:25.104173 14724 raft_consensus.cc:504] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:25.104857 14724 leader_election.cc:288] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [CANDIDATE]: Term 1 election: Requested vote from peers 61e33cc2a3b04d59b3deabb7f908a7c9 (127.2.205.131:42255), 493a97e82ba943b9828f790f5d593d71 (127.2.205.129:44459)
I20260929 03:03:25.105350 14653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31397c9b3f54ce9a53cefc43f1871b0" candidate_uuid: "a61d1f9e37f64b78b642b365009d407b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9"
I20260929 03:03:25.105448 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31397c9b3f54ce9a53cefc43f1871b0" candidate_uuid: "a61d1f9e37f64b78b642b365009d407b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493a97e82ba943b9828f790f5d593d71"
I20260929 03:03:25.105549 14653 raft_consensus.cc:3037] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:03:25.105700 14376 raft_consensus.cc:3037] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:03:25.107942 14653 raft_consensus.cc:2454] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61d1f9e37f64b78b642b365009d407b in term 1.
I20260929 03:03:25.107954 14376 raft_consensus.cc:2454] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61d1f9e37f64b78b642b365009d407b in term 1.
I20260929 03:03:25.108415 14452 leader_election.cc:302] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 493a97e82ba943b9828f790f5d593d71, 61e33cc2a3b04d59b3deabb7f908a7c9, a61d1f9e37f64b78b642b365009d407b; no voters:
I20260929 03:03:25.108812 14724 raft_consensus.cc:2781] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:03:25.109238 14724 raft_consensus.cc:686] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 1 LEADER]: Becoming Leader. State: Replica: a61d1f9e37f64b78b642b365009d407b, State: Running, Role: LEADER
I20260929 03:03:25.109788 14724 consensus_queue.cc:237] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } }
I20260929 03:03:25.114513 14208 catalog_manager.cc:5495] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b reported cstate change: term changed from 0 to 1, leader changed from <none> to a61d1f9e37f64b78b642b365009d407b (127.2.205.130). New cstate: current_term: 1 leader_uuid: "a61d1f9e37f64b78b642b365009d407b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61d1f9e37f64b78b642b365009d407b" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 37915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 } health_report { overall_health: UNKNOWN } } }
W20260929 03:03:25.152987 14422 tablet.cc:2369] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:03:25.175854 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.170599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.2.205.130:37915)
0929 03:03:25.170789 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.170794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.170812 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:03:25.170962 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.170965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.171051 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.171188 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:03:25.171605 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.172331 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.173845 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.174254 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.174350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.174794 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:03:25.175308 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:03:25.175416 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.175541 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.175608 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:03:25.184199 14376 raft_consensus.cc:1261] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 1 FOLLOWER]: Refusing update from remote peer a61d1f9e37f64b78b642b365009d407b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:03:25.184199 14653 raft_consensus.cc:1261] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 1 FOLLOWER]: Refusing update from remote peer a61d1f9e37f64b78b642b365009d407b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:03:25.185168 14728 consensus_queue.cc:1035] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [LEADER]: Connected to new peer: Peer: permanent_uuid: "493a97e82ba943b9828f790f5d593d71" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:03:25.185500 14724 consensus_queue.cc:1035] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e33cc2a3b04d59b3deabb7f908a7c9" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 42255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:03:25.205868 14731 mvcc.cc:205] Tried to move back new op lower bound from 7334506517224493056 to 7334506516934053888. Current Snapshot: MvccSnapshot[applied={T|T < 7334506517224493056 or (T in {7334506517224493056})}]
I20260929 03:03:25.214774 14208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:03:25.215062 14208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:03:25.221235 14208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad6f0cafd77e45eab222217c1f19a0d2: Sending DeleteTablet for 3 replicas of tablet b31397c9b3f54ce9a53cefc43f1871b0
I20260929 03:03:25.222473 14356 tablet_service.cc:1514] Processing DeleteTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:03:25 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:55944
I20260929 03:03:25.223417 14741 tablet_replica.cc:330] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: stopping tablet replica
I20260929 03:03:25.223717 14633 tablet_service.cc:1514] Processing DeleteTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:03:25 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:60510
I20260929 03:03:25.223996 14741 raft_consensus.cc:2229] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:03:25.224491 14741 pending_rounds.cc:62] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Trying to abort 2 pending ops.
I20260929 03:03:25.224668 14741 pending_rounds.cc:68] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334506517224493056 op_type: WRITE_OP write_request { tablet_id: "b31397c9b3f54ce9a53cefc43f1871b0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\251\327\354\325\006\"8\n\ntest-admin\022*\n 1e3645c32d8b4470b9ef4d0d88cbc4fa\020\001\030\001 \001(\001" signature: "<redacted>""?Y\023#\240\364\023\231\363\2073\210\303\027\267\027\365\251\\\367\247\305\260\216\227\304\336\177\261\364\351oj\300\355\367\347@A\330\254\223\375\200\212i\213\267\347\234\3458A\265)\355\201\3752\273\234l\2152" signing_key_seq_num: 0 } } request_id { client_id: "b3a1ebacc29b42e98f5b407ba07c66a2" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260929 03:03:25.226581 14741 pending_rounds.cc:68] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334506516934053888 op_type: NO_OP noop_request { }
I20260929 03:03:25.226848 14741 raft_consensus.cc:2866] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 03:03:25.227084 14741 raft_consensus.cc:2258] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:03:25.227895 14742 tablet_replica.cc:330] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: stopping tablet replica
I20260929 03:03:25.228351 14742 raft_consensus.cc:2229] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:03:25.228427 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 14288
W20260929 03:03:25.228736 14452 consensus_peers.cc:487] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b -> Peer 493a97e82ba943b9828f790f5d593d71 (127.2.205.129:44459): Couldn't send request to peer 493a97e82ba943b9828f790f5d593d71. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:03:25.229393 14499 tablet_service.cc:1514] Processing DeleteTablet for tablet b31397c9b3f54ce9a53cefc43f1871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:03:25 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:42376
I20260929 03:03:25.229909 14742 raft_consensus.cc:2258] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:03:25.230783 14741 ts_tablet_manager.cc:1905] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:03:25.237183 14742 ts_tablet_manager.cc:1905] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:03:25.238133 14741 ts_tablet_manager.cc:1918] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:03:25.238404 14741 log.cc:1196] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/b31397c9b3f54ce9a53cefc43f1871b0
I20260929 03:03:25.238970 14741 ts_tablet_manager.cc:1939] T b31397c9b3f54ce9a53cefc43f1871b0 P 493a97e82ba943b9828f790f5d593d71: Deleting consensus metadata
I20260929 03:03:25.241176 14192 catalog_manager.cc:4841] TS 493a97e82ba943b9828f790f5d593d71 (127.2.205.129:44459): tablet b31397c9b3f54ce9a53cefc43f1871b0 (table test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]) successfully deleted
I20260929 03:03:25.243911 14742 ts_tablet_manager.cc:1918] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:03:25.244074 14742 log.cc:1196] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/b31397c9b3f54ce9a53cefc43f1871b0
I20260929 03:03:25.244460 14742 ts_tablet_manager.cc:1939] T b31397c9b3f54ce9a53cefc43f1871b0 P 61e33cc2a3b04d59b3deabb7f908a7c9: Deleting consensus metadata
I20260929 03:03:25.246455 14193 catalog_manager.cc:4841] TS 61e33cc2a3b04d59b3deabb7f908a7c9 (127.2.205.131:42255): tablet b31397c9b3f54ce9a53cefc43f1871b0 (table test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]) successfully deleted
I20260929 03:03:25.247519 14743 tablet_replica.cc:330] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: stopping tablet replica
I20260929 03:03:25.247916 14743 raft_consensus.cc:2229] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:03:25.248585 14743 raft_consensus.cc:2258] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:03:25.250276 14743 ts_tablet_manager.cc:1905] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:03:25.255695 14743 ts_tablet_manager.cc:1918] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:03:25.255859 14743 log.cc:1196] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/b31397c9b3f54ce9a53cefc43f1871b0
Sep 29 03:03:25 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:03:25.256366 14743 ts_tablet_manager.cc:1939] T b31397c9b3f54ce9a53cefc43f1871b0 P a61d1f9e37f64b78b642b365009d407b: Deleting consensus metadata
I20260929 03:03:25.257926 14193 catalog_manager.cc:4841] TS a61d1f9e37f64b78b642b365009d407b (127.2.205.130:37915): tablet b31397c9b3f54ce9a53cefc43f1871b0 (table test-table [id=1e3645c32d8b4470b9ef4d0d88cbc4fa]) successfully deleted
I20260929 03:03:25.263118 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.230208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.2.205.129:44459)
0929 03:03:25.230388 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.230395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.245409 (+ 15014us) server_negotiation.cc:407] Connection header received
0929 03:03:25.245588 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.245593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.245693 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.245822 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:25.247750 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.248901 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.250807 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.251382 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.251525 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.257239 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:25.257262 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:25.257280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:25.257322 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:25.260183 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.261092 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.261117 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.261120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.261278 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.261664 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.261668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.261670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.262016 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:25.262237 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.262627 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.262789 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:03:25.263739 14416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:03:25.413908 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 14288
I20260929 03:03:25.429868 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 14427
Sep 29 03:03:25 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:03:25.448040 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.430703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.2.205.130:37915)
0929 03:03:25.430888 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.430896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.430954 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:03:25.431152 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.431156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.431229 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.431345 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:25.433059 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.433901 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.435761 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.436177 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.436272 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.441635 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:25.441653 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:25.441674 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:25.441719 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:25.445035 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.445919 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.445927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.445929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.446090 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.446475 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.446478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.446480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.446832 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:25.447071 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.447453 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.447688 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:03:25.448769 14559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:03:25.574781 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 14427
I20260929 03:03:25.590385 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 14570
Sep 29 03:03:25 dist-test-slave-530n krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:03:25.606024 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:03:25.591177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.2.205.131:42255)
0929 03:03:25.591365 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:03:25.591371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:03:25.591396 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:03:25.591604 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:03:25.591608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:03:25.591671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:03:25.591756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:03:25.593232 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.593944 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.595350 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:03:25.595735 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:03:25.595853 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:03:25.600342 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:03:25.600361 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:03:25.600378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:03:25.600424 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:03:25.603183 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.603928 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.603937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.603940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.604089 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:03:25.604469 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:03:25.604473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:03:25.604476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:03:25.604917 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:03:25.605179 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 03:03:25.605487 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 03:03:25.605688 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:03:25.606639 14693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:03:25.694697 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 14570
I20260929 03:03:25.707955 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 14176
I20260929 03:03:25.709167 14238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:03:25.820336 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 14176
I20260929 03:03:25.840169 2870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:03:26.175868 2870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:03:26.176070 2870 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:03:26.521682 2870 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:03:26.521 UTC [13762] LOG: received smart shutdown request
2026-09-29 03:03:26.525 UTC [13762] LOG: background worker "logical replication launcher" (PID 13771) exited with exit code 1
2026-09-29 03:03:26.525 UTC [13766] LOG: shutting down
2026-09-29 03:03:26.670 UTC [13762] LOG: database system is shut down
2026-09-29T03:03:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47757 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:03:26 dist-test-slave-530n krb5kdc[14757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:03:26 dist-test-slave-530n krb5kdc[14757](info): set up 2 sockets
Sep 29 03:03:26 dist-test-slave-530n krb5kdc[14757](info): commencing operation
krb5kdc: starting...
W20260929 03:03:27.797343 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:03:27 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:03:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangeradmin_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangeradmin_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangerlookup_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangerlookup_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
I20260929 03:03:28.097425 2870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskg3SEC6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/postgres -l logfile start
2026-09-29 03:03:30.414 UTC [14787] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:03:30.414 UTC [14787] LOG: listening on IPv4 address "127.2.205.148", port 46803
2026-09-29 03:03:30.417 UTC [14787] LOG: listening on Unix socket "/tmp/.s.PGSQL.46803"
2026-09-29 03:03:30.438 UTC [14790] LOG: database system was shut down at 2026-09-29 03:03:29 UTC
2026-09-29 03:03:30.442 UTC [14787] LOG: database system is ready to accept connections
I20260929 03:03:31.337056 2870 mini_postgres.cc:96] Postgres bound to 46803
2026-09-29 03:03:31.352 UTC [14798] FATAL: database "slave" does not exist
127.2.205.148:46803 - accepting connections
I20260929 03:03:31.353341 2870 mini_ranger.cc:161] Starting Ranger...
I20260929 03:03:31.376000 2870 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:03:31.811504 2870 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:03:31.811604 2870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin
2026-09-29 03:03:32,091 [I] DB FLAVOR :POSTGRES
2026-09-29 03:03:32,091 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:03:32,091 [I] Checking connection..
2026-09-29 03:03:32,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:03:32,338 [I] Checking connection passed.
2026-09-29 03:03:32,338 [I] --------- Verifying version history table ---------
2026-09-29 03:03:32,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:03:32,603 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:03:32,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:03:32,849 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:03:32,850 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:03:32,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:03:33.080 UTC [14862] WARNING: there is no transaction in progress
2026-09-29 03:03:33,087 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:03:33,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:03:33,343 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:03:33,344 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:03:33,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:03:33,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:03:33,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-530n.c.gcp-upstream.internal','N') ;"
2026-09-29 03:03:34,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:03:34,323 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:03:34,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:03:34,587 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:03:34,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:03:34,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:03:35,062 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:03:35,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:03:36.031 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.047 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.063 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.079 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.095 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.114 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.173 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.180 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.195 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.206 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.217 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.231 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.241 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.248 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.254 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36.261 UTC [14970] WARNING: there is no transaction in progress
2026-09-29 03:03:36,892 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:03:36,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:03:37,116 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:03:37,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:03:37,358 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:03:37,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:03:37,591 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:03:37,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:03:37,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:03:38,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-530n.c.gcp-upstream.internal';"
2026-09-29 03:03:38,344 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:03:38,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:03:38,577 [I] DB_PATCHES have already been applied
I20260929 03:03:38.583426 2870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskg3SEC6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:03:38.583518 2870 mini_ranger.cc:193] Using host: 127.2.205.148
I20260929 03:03:38.596431 2870 mini_ranger.cc:227] Ranger admin URL: http://127.2.205.148:40571
Sep 29, 2026 3:03:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:03:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:03:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:03:39.631244 2870 mini_ranger.cc:160] Time spent starting Ranger: real 8.278s user 0.001s sys 0.036s
Sep 29, 2026 3:04:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:04:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 3:04:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:04:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:04:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:04:01.618563 2870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.205.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.205.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangerkms_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.205.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/rangerkms_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.205.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.205.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.205.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/HTTP_127.2.205.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:04:01 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651041, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:04:01.774758 2870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:04:01.794021 2870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:04:02.634447 2870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:04:02.634545 2870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms
I20260929 03:04:02.634585 2870 mini_ranger_kms.cc:227] Using postgres at 127.2.205.148:46803
2026-09-29 03:04:02,978 [I] DB FLAVOR :POSTGRES
2026-09-29 03:04:02,978 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:04:02,979 [I] Checking connection
2026-09-29 03:04:02,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:04:03,245 [I] connection success
2026-09-29 03:04:03,245 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:04:03,245 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:04:03,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:04:03,511 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:04:03,511 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:04:03,511 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:04:03,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.205.148:46803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:04:03,791 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:04:04.240489 2870 mini_ranger_kms.cc:310] Created kms service
I20260929 03:04:04.433555 2870 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:04:04.481009 2870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:04:04 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651041, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:04.893996 2870 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:04:04.894320 2870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskg3SEC6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskg3SEC6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:04:04.894378 2870 mini_ranger_kms.cc:240] Using host: 127.2.205.148
I20260929 03:04:04.911250 2870 mini_ranger_kms.cc:276] Ranger KMS PID: 15150
I20260929 03:04:04.911391 2870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.205.148:40411
Sep 29, 2026 3:04:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:04:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:04:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:04:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:04:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:04:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:04:06.963238 2870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.188s user 0.058s sys 0.304s
I20260929 03:04:06.963454 2870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:04:06.963606 2870 mini_ranger_kms.cc:401] 127.2.205.148:40411/kms/v1/keys
Sep 29 03:04:07 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:04:07 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
Sep 29 03:04:07 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.205.148@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
Sep 29 03:04:10 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:04:10.183024 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38231
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:38231
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:10.520574 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:10.520965 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:10.521106 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:10.528657 15201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:10.528851 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:10.528936 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:10.529014 15201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:10.529078 15201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:10.539788 15201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.205.148:40411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:38231
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:10.541240 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:10.542534 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:10.554388 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:10.554844 15201 server_base.cc:1034] running on GCE node
W20260929 03:04:10.555001 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:10.555045 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:10.556306 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:10.558068 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:10.559336 15201 hybrid_clock.cc:648] HybridClock initialized: now 1790651050559276 us; error 54 us; skew 500 ppm
I20260929 03:04:10.559731 15201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:10 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:10.563459 15201 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:10.564965 15201 webserver.cc:460] Webserver started at http://127.2.205.190:42033/ using document root <none> and password file <none>
I20260929 03:04:10.565603 15201 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:10.565699 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:10.565997 15201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:04:10 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:10.615410 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "c4de619118d84f198369ac0d24308c17"
format_stamp: "Formatted at 2026-09-29 03:04:10 on dist-test-slave-530n"
server_key: "a07da48f867e4ddb12ea9ae293c46d4a"
server_key_iv: "c2c0aad8b987740a942d4653762c2c06"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:10.616247 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "c4de619118d84f198369ac0d24308c17"
format_stamp: "Formatted at 2026-09-29 03:04:10 on dist-test-slave-530n"
server_key: "a07da48f867e4ddb12ea9ae293c46d4a"
server_key_iv: "c2c0aad8b987740a942d4653762c2c06"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:10.621646 15201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 03:04:10.638540 15216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:10.640122 15201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:04:10.640303 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "c4de619118d84f198369ac0d24308c17"
format_stamp: "Formatted at 2026-09-29 03:04:10 on dist-test-slave-530n"
server_key: "a07da48f867e4ddb12ea9ae293c46d4a"
server_key_iv: "c2c0aad8b987740a942d4653762c2c06"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:10.640528 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:10.661612 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:10.665690 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:10.665997 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:10.684506 15201 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:38231
I20260929 03:04:10.684507 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:38231 every 8 connection(s)
I20260929 03:04:10.685961 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:10.688887 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15201
I20260929 03:04:10.689113 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:04:10.689914 2870 external_mini_cluster.cc:1428] Setting key 8a578ea5ac5467f138c0b0c8b9ee4760
I20260929 03:04:10.690169 15268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:10.707067 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Bootstrap starting.
I20260929 03:04:10.710599 15268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:10.712231 15268 log.cc:824] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:10.715489 15268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: No bootstrap required, opened a new log
Sep 29 03:04:10 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:10.725170 15268 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:10.725705 15268 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:10.725831 15268 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4de619118d84f198369ac0d24308c17, State: Initialized, Role: FOLLOWER
I20260929 03:04:10.726504 15268 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:10.726747 15268 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:10.726917 15268 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:10.727089 15268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:10.729934 15268 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:10.730582 15268 leader_election.cc:302] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4de619118d84f198369ac0d24308c17; no voters:
I20260929 03:04:10.733146 15268 leader_election.cc:288] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:04:10.733170 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:10.692055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.2.205.190:38231)
0929 03:04:10.706945 (+ 14890us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:10.706964 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:10.707020 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:04:10.709540 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:10.709573 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:10.710164 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:10.710674 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:10.712363 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:10.714544 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:10.716480 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:10.717704 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:10.717963 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:10.723252 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:10.723271 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:10.723305 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:10.723369 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:10.727526 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:10.728526 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:10.728536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:10.728547 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:10.728808 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:10.729293 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:10.729298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:10.729301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:10.730137 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:10.730442 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:10.730976 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 03:04:10.731458 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14589,"thread_start_us":16682,"threads_started":1}
I20260929 03:04:10.733207 15273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:10.740032 15268 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:10.740599 15273 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 LEADER]: Becoming Leader. State: Replica: c4de619118d84f198369ac0d24308c17, State: Running, Role: LEADER
I20260929 03:04:10.741338 15273 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:10.744496 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4de619118d84f198369ac0d24308c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } } }
I20260929 03:04:10.744957 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:10.750756 15275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4de619118d84f198369ac0d24308c17. Latest consensus state: current_term: 1 leader_uuid: "c4de619118d84f198369ac0d24308c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } } }
I20260929 03:04:10.751312 15275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:10.751392 15278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:10.759940 15278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:10.760663 15268 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:04:10.788430 15278 catalog_manager.cc:1347] Generated new cluster ID: d129cfaebf1e44b0be11662c73fbecb0
I20260929 03:04:10.788625 15278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:10.820185 15278 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:04:10.822391 15278 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:10.845901 15278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Generated new TSK 0
I20260929 03:04:10.847132 15278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:04:11.629110 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:11 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:04:12.051964 15313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:12.052418 15313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:12.052583 15313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:12.063948 15313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:12.064080 15313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:12.064320 15313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:12.079883 15313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:12.081578 15313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:12.083283 15313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:12.108974 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:12.110873 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:12.126441 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:12.126962 15313 server_base.cc:1034] running on GCE node
I20260929 03:04:12.127731 15313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:12.129662 15313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:12.130985 15313 hybrid_clock.cc:648] HybridClock initialized: now 1790651052130905 us; error 65 us; skew 500 ppm
I20260929 03:04:12.131364 15313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:12 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:12.135347 15313 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:12.137259 15313 webserver.cc:460] Webserver started at http://127.2.205.129:41109/ using document root <none> and password file <none>
I20260929 03:04:12.138134 15313 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:12.138267 15313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:12.138607 15313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:04:12 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:12.158488 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "84dada6457e24f3cac5f4e3bedbb6991"
format_stamp: "Formatted at 2026-09-29 03:04:12 on dist-test-slave-530n"
server_key: "d4bed0994d4d75a2ea42d04eebea1657"
server_key_iv: "ef4fabb091d0a39236db460aa1862668"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:12.159713 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "84dada6457e24f3cac5f4e3bedbb6991"
format_stamp: "Formatted at 2026-09-29 03:04:12 on dist-test-slave-530n"
server_key: "d4bed0994d4d75a2ea42d04eebea1657"
server_key_iv: "ef4fabb091d0a39236db460aa1862668"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:12.179513 15313 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.008s sys 0.004s
I20260929 03:04:12.229483 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:12.231838 15313 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.009s sys 0.011s
I20260929 03:04:12.232123 15313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "84dada6457e24f3cac5f4e3bedbb6991"
format_stamp: "Formatted at 2026-09-29 03:04:12 on dist-test-slave-530n"
server_key: "d4bed0994d4d75a2ea42d04eebea1657"
server_key_iv: "ef4fabb091d0a39236db460aa1862668"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:12.232849 15313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:12.308589 15313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:12.313712 15313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:12.314029 15313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:12.315728 15313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:12.318305 15313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:12.318435 15313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 03:04:12.318576 15313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:12.318653 15313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:04:12 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:12.414520 15313 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:39603
I20260929 03:04:12.414595 15445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:39603 every 8 connection(s)
I20260929 03:04:12.416199 15313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:12.425431 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15313
I20260929 03:04:12.425645 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:04:12.426435 2870 external_mini_cluster.cc:1428] Setting key fe94fab367675f88c068fa64c1c03c7d
Sep 29 03:04:12 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:12.454934 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:12.420634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:34945 (local address 127.2.205.190:38231)
0929 03:04:12.425425 (+ 4791us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:12.425432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:12.436062 (+ 10630us) server_negotiation.cc:407] Connection header received
0929 03:04:12.436131 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:12.436137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:12.436232 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:12.436324 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:12.438522 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:12.439712 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:12.442338 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:12.443015 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:12.443113 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:12.447749 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:12.447755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:12.447759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:12.447798 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:12.451071 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:12.451961 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:12.451967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:12.451971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:12.452124 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:12.452655 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:12.452660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:12.452664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:12.453049 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:12.453178 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:12.454258 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 03:04:12.454496 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4651,"thread_start_us":7234,"threads_started":1}
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:04:12.457792 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:12.420006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.129:34945)
0929 03:04:12.433051 (+ 13045us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x60c00006c080. stack: 00007f8ed7c7166e 00007f8ed7c71430 00007f8ed6cf85ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f8eeab54775 00007f8ed7cb71d3 00007f8ed7cba91b 00007f8ed7cba798 00000000009edb7c 00007f8ed7c95eae 00007f8edaf6f6da 00007f8ed3f0561e
0929 03:04:12.433303 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:12.433353 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:12.435673 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:12.436782 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:12.436805 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:12.437411 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:12.438315 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:12.438341 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:12.440017 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:12.440026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:12.442171 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:12.442187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:12.443353 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:12.443358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:12.443738 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:12.444570 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:12.444613 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:12.447490 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:12.451211 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:12.451220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:12.451239 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:12.451811 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:12.452260 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:12.452265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:12.452269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:12.452528 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:12.453268 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:12.453276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:12.453824 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 03:04:12.455543 (+ 1719us) client_negotiation.cc:240] Negotiation successful
0929 03:04:12.456145 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4980,"spinlock_wait_cycles":6873344,"thread_start_us":4784,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
I20260929 03:04:12.459911 15446 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:12.460248 15446 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:12.461122 15446 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:12.466118 15233 ts_manager.cc:194] Registered new tserver with Master: 84dada6457e24f3cac5f4e3bedbb6991 (127.2.205.129:39603)
I20260929 03:04:12.468189 15233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:34945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:04:12.491969 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:12 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
W20260929 03:04:12.968129 15452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:12.968647 15452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:12.968860 15452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:12.980422 15452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:12.980643 15452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:12.980894 15452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:12.994517 15452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:12.995838 15452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:12.997318 15452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:13.009990 15467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:13.010028 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:13.010986 15452 server_base.cc:1034] running on GCE node
W20260929 03:04:13.011318 15469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:13.011909 15452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:13.012876 15452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:13.014082 15452 hybrid_clock.cc:648] HybridClock initialized: now 1790651053014034 us; error 44 us; skew 500 ppm
I20260929 03:04:13.014374 15452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:13.017373 15452 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:13.018751 15452 webserver.cc:460] Webserver started at http://127.2.205.130:46021/ using document root <none> and password file <none>
I20260929 03:04:13.019394 15452 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:13.019505 15452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:13.019771 15452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:13.037479 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "adbe6c3527134a07ae53f5a6ea1707c7"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "50dec83418cc65de410c26f495518650"
server_key_iv: "73107310942a0e29ee395ab101d9b1ae"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.038661 15452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "adbe6c3527134a07ae53f5a6ea1707c7"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "50dec83418cc65de410c26f495518650"
server_key_iv: "73107310942a0e29ee395ab101d9b1ae"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.045903 15452 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 03:04:13.066905 15476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.068696 15452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 03:04:13.068966 15452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "adbe6c3527134a07ae53f5a6ea1707c7"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "50dec83418cc65de410c26f495518650"
server_key_iv: "73107310942a0e29ee395ab101d9b1ae"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.069226 15452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:13.111717 15452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:13.117044 15452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:13.117370 15452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:13.118544 15452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:13.120297 15452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:13.120390 15452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.120560 15452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:13.120641 15452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.164580 15452 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:40243
I20260929 03:04:13.164605 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:40243 every 8 connection(s)
I20260929 03:04:13.166330 15452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:13.175536 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15452
I20260929 03:04:13.175711 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:04:13.176272 2870 external_mini_cluster.cc:1428] Setting key 7af4e21e32e64ff46b260cdebf7bac7a
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:13.201624 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.170476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:42165 (local address 127.2.205.190:38231)
0929 03:04:13.182891 (+ 12415us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:13.182896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:13.182913 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:04:13.182960 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:13.182965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:13.183059 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:13.183172 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:13.188203 (+ 5031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.189010 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.190662 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.191057 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.191164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.194886 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:13.194894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:13.194899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:13.194954 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:13.198157 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.198995 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.199000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.199004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.199150 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.199620 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.199625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.199628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.199985 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:13.200134 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:13.201030 (+ 896us) server_negotiation.cc:299] Negotiation successful
0929 03:04:13.201254 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12288,"thread_start_us":12175,"threads_started":1}
I20260929 03:04:13.203250 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.169852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.130:42165)
0929 03:04:13.178853 (+ 9001us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:13.178931 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:13.181533 (+ 2602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:13.183531 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:13.183545 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:13.185077 (+ 1532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:13.185989 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:13.186019 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.189182 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.189188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:13.190496 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:13.190515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.191286 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.191294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:13.191529 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.192194 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:13.192218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:13.194620 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:13.198301 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:13.198311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:13.198328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:13.198852 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:13.199259 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:13.199264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:13.199267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:13.199506 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:13.200120 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:13.200128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:13.200529 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 03:04:13.200869 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 03:04:13.201391 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8643,"thread_start_us":8361,"threads_started":1}
I20260929 03:04:13.204825 15589 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:13.205194 15589 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:13.206059 15589 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:04:13.207667 15233 ts_manager.cc:194] Registered new tserver with Master: adbe6c3527134a07ae53f5a6ea1707c7 (127.2.205.130:40243)
I20260929 03:04:13.208806 15233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:42165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:04:13.234203 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:13.471895 15446 heartbeater.cc:503] Master 127.2.205.190:38231 was elected leader, sending a full tablet report...
W20260929 03:04:13.546845 15595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:13.547102 15595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:13.547235 15595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:13.554554 15595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:13.554657 15595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:13.554821 15595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:13.564432 15595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:13.565729 15595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:13.567042 15595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:13.580086 15600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:13.580020 15601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:13.580052 15603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:13.580926 15595 server_base.cc:1034] running on GCE node
I20260929 03:04:13.581550 15595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:13.582509 15595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:13.583706 15595 hybrid_clock.cc:648] HybridClock initialized: now 1790651053583648 us; error 47 us; skew 500 ppm
I20260929 03:04:13.584035 15595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:13.587031 15595 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:13.588455 15595 webserver.cc:460] Webserver started at http://127.2.205.131:35373/ using document root <none> and password file <none>
I20260929 03:04:13.589200 15595 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:13.589305 15595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:13.589598 15595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:13.606453 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "85a6574af0c345399d55a84274b7fafa"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "adf88dc1082ac19aed0828826358ff4a"
server_key_iv: "f34f3ef5d85251f450b62a1bdccf123e"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.607565 15595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "85a6574af0c345399d55a84274b7fafa"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "adf88dc1082ac19aed0828826358ff4a"
server_key_iv: "f34f3ef5d85251f450b62a1bdccf123e"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.613974 15595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 03:04:13.628667 15610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.630370 15595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:04:13.630533 15595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "85a6574af0c345399d55a84274b7fafa"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "adf88dc1082ac19aed0828826358ff4a"
server_key_iv: "f34f3ef5d85251f450b62a1bdccf123e"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:13.630767 15595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:13.643299 15595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:13.647032 15595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:13.647295 15595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:13.648123 15595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:13.649398 15595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:13.649471 15595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.649575 15595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:13.649624 15595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:13.678447 15595 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41717
I20260929 03:04:13.678476 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41717 every 8 connection(s)
I20260929 03:04:13.679630 15595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:13.686898 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15595
I20260929 03:04:13.687060 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:04:13.687516 2870 external_mini_cluster.cc:1428] Setting key 87d2a7eb2200ebb0c72202a84972d560
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:13.698894 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.681981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:54021 (local address 127.2.205.190:38231)
0929 03:04:13.682144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:13.682148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:13.683318 (+ 1170us) server_negotiation.cc:407] Connection header received
0929 03:04:13.685119 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:13.685125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:13.685240 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:13.685328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:13.686711 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.687822 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.689796 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.690434 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.690533 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.693739 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:13.693743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:13.693746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:13.693775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:13.696112 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.696781 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.696787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.696790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.696920 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.697316 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.697320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.697322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.697576 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:13.697716 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:13.698340 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 03:04:13.698575 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:04:13.699800 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.682491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.131:54021)
0929 03:04:13.683170 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:13.683213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:13.684845 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:13.685564 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:13.685572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:13.685977 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:13.686531 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:13.686552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.687972 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.687977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:13.689644 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:13.689659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.690816 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.690821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:13.691035 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.691621 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:13.691642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:13.693542 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:13.696233 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:13.696242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:13.696257 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:13.696648 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:13.697018 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:13.697023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:13.697027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:13.697211 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:13.697691 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:13.697699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:13.698001 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 03:04:13.698228 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 03:04:13.698778 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":268,"threads_started":1}
I20260929 03:04:13.701164 15723 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:13.701440 15723 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:13.702275 15723 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:13.703770 15233 ts_manager.cc:194] Registered new tserver with Master: 85a6574af0c345399d55a84274b7fafa (127.2.205.131:41717)
I20260929 03:04:13.704591 15233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:54021
I20260929 03:04:13.715380 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:04:13.717087 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15313
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:04:13.732928 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.717936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.2.205.129:39603)
0929 03:04:13.718533 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:13.718543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:13.718573 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:04:13.718711 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:13.718719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:13.718942 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:13.719083 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:13.720500 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.721554 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.723045 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.723601 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.723713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.727754 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:13.727766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:13.727780 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:13.727816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:13.730266 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.730901 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.730908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.730910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.731042 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.731377 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.731387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.731389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.731706 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:13.731977 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:13.732289 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 03:04:13.732566 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":261,"threads_started":1}
I20260929 03:04:13.734107 15441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:13.815941 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15313
I20260929 03:04:13.827965 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15452
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:04:13.843293 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.828788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.2.205.130:40243)
0929 03:04:13.829280 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:13.829287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:13.829310 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:04:13.829369 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:13.829376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:13.829573 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:13.829710 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:13.830915 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.831903 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.833233 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.833759 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.833878 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.838030 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:13.838041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:13.838056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:13.838094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:13.840546 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.841266 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.841273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.841275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.841404 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.841754 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.841758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.841760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.842099 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:13.842372 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:13.842691 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 03:04:13.842971 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260929 03:04:13.844533 15584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:13.923750 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15452
I20260929 03:04:13.939734 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15595
Sep 29 03:04:13 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:04:13.955441 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:13.940731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.2.205.131:41717)
0929 03:04:13.941467 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:13.941475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:13.941502 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:04:13.941558 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:13.941564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:13.941831 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:13.941991 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:13.943189 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.944161 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.945642 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:13.946234 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:13.946343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:13.950346 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:13.950357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:13.950371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:13.950405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:13.953052 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.953681 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.953687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.953690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.953810 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:13.954113 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:13.954116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:13.954118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:13.954449 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:13.954662 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:13.954927 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:04:13.955110 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
I20260929 03:04:13.956646 15718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:14.036675 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15595
I20260929 03:04:14.048882 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15201
I20260929 03:04:14.050060 15263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:14.154405 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15201
I20260929 03:04:14.171847 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38231
--webserver_interface=127.2.205.190
--webserver_port=42033
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:38231
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:14.476943 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:14.477234 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:14.477375 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:14.484436 15734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:14.484546 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:14.484624 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:14.484699 15734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:14.484776 15734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:14.494491 15734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:38231
--ranger_config_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=42033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:14.495692 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:14.497011 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:14.507869 15740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:14.507936 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:14.507953 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:14.508621 15734 server_base.cc:1034] running on GCE node
I20260929 03:04:14.509344 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:14.510653 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:14.511866 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790651054511804 us; error 59 us; skew 500 ppm
I20260929 03:04:14.512182 15734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:14 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:14.514989 15734 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:14.516273 15734 webserver.cc:460] Webserver started at http://127.2.205.190:42033/ using document root <none> and password file <none>
I20260929 03:04:14.516901 15734 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:14.517010 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:04:14 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:14.562572 15734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 03:04:14.565527 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:14.567059 15734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:04:14.567220 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "c4de619118d84f198369ac0d24308c17"
format_stamp: "Formatted at 2026-09-29 03:04:10 on dist-test-slave-530n"
server_key: "a07da48f867e4ddb12ea9ae293c46d4a"
server_key_iv: "c2c0aad8b987740a942d4653762c2c06"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:14.568183 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:14.591569 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:14.595191 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:14.595417 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:14.611950 15734 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:38231
I20260929 03:04:14.611984 15801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:38231 every 8 connection(s)
I20260929 03:04:14.613344 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:14.619853 15802 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:04:14.621326 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15734
I20260929 03:04:14.622303 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:39603
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:38231
--webserver_port=41109
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:14.623989 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Bootstrap starting.
I20260929 03:04:14.643224 15802 log.cc:824] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:14.651803 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:04:14.652264 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Bootstrap complete.
I20260929 03:04:14.660599 15802 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:14.661429 15802 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4de619118d84f198369ac0d24308c17, State: Initialized, Role: FOLLOWER
I20260929 03:04:14.662002 15802 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:14.662194 15802 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:14.662299 15802 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:14.662453 15802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:04:14.666102 15802 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:14.666568 15802 leader_election.cc:302] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4de619118d84f198369ac0d24308c17; no voters:
I20260929 03:04:14.667145 15802 leader_election.cc:288] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:04:14.667284 15806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:04:14.668522 15806 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [term 2 LEADER]: Becoming Leader. State: Replica: c4de619118d84f198369ac0d24308c17, State: Running, Role: LEADER
I20260929 03:04:14.669127 15806 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } }
I20260929 03:04:14.669392 15802 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:14.671526 15807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c4de619118d84f198369ac0d24308c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } } }
I20260929 03:04:14.671909 15807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:14.674117 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4de619118d84f198369ac0d24308c17. Latest consensus state: current_term: 2 leader_uuid: "c4de619118d84f198369ac0d24308c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4de619118d84f198369ac0d24308c17" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 38231 } } }
I20260929 03:04:14.674502 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:14.676074 15811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:14.682405 15811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:14.683169 15802 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:04:14.683876 15811 catalog_manager.cc:1259] Loaded cluster ID: d129cfaebf1e44b0be11662c73fbecb0
I20260929 03:04:14.683962 15811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:14.688176 15811 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:14.689661 15811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Loaded TSK: 0
I20260929 03:04:14.691293 15811 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:04:15.011516 15804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:15.012001 15804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:15.012224 15804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:15.021631 15804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:15.021757 15804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:15.021919 15804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:15.035887 15804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:39603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=41109
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:15.037699 15804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:15.039472 15804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:15.068012 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:15.068261 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:15.075290 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:15.076253 15804 server_base.cc:1034] running on GCE node
I20260929 03:04:15.077204 15804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:15.078361 15804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:15.079622 15804 hybrid_clock.cc:648] HybridClock initialized: now 1790651055079567 us; error 47 us; skew 500 ppm
I20260929 03:04:15.079947 15804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:15.084777 15804 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:15.093814 15804 webserver.cc:460] Webserver started at http://127.2.205.129:41109/ using document root <none> and password file <none>
I20260929 03:04:15.094595 15804 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:15.094731 15804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:15.127713 15804 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.000s
I20260929 03:04:15.141415 15850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:15.143229 15804 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.007s
I20260929 03:04:15.143435 15804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "84dada6457e24f3cac5f4e3bedbb6991"
format_stamp: "Formatted at 2026-09-29 03:04:12 on dist-test-slave-530n"
server_key: "d4bed0994d4d75a2ea42d04eebea1657"
server_key_iv: "ef4fabb091d0a39236db460aa1862668"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:15.144482 15804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:15.187789 15804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:15.193331 15804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:15.193712 15804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:15.194870 15804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:15.202551 15804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:15.202703 15804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:15.202867 15804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:15.202965 15804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:15.307323 15804 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:39603
I20260929 03:04:15.307380 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:39603 every 8 connection(s)
I20260929 03:04:15.308723 15804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:15.312496 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15804
I20260929 03:04:15.313341 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:40243
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:38231
--webserver_port=46021
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:15.371266 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:15.311933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.129:58217)
0929 03:04:15.333413 (+ 21480us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:15.333480 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:15.335881 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:15.350518 (+ 14637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:15.350531 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:15.351184 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:15.352037 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:15.352063 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.354748 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.354754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:15.356718 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:15.356733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.358132 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.358138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:15.358389 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:15.359092 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:15.359121 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:15.361665 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:15.366117 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:15.366126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:15.366140 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:15.366732 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:15.367273 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:15.367278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:15.367282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:15.367626 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:15.368594 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:15.368602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:15.369008 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 03:04:15.369433 (+ 425us) client_negotiation.cc:240] Negotiation successful
0929 03:04:15.369949 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21145,"thread_start_us":20912,"threads_started":1}
I20260929 03:04:15.374323 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:15.312009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:58217 (local address 127.2.205.190:38231)
0929 03:04:15.348522 (+ 36513us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:15.348543 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:15.348599 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:04:15.349084 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:15.349106 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:15.349634 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:15.352306 (+ 2672us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:15.352683 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.354572 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.356943 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.357887 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.358827 (+ 940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:15.361946 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:15.361957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:15.361976 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:15.362032 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:15.365941 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:15.366928 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:15.366937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:15.366944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:15.367132 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:15.367784 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:15.367788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:15.367791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:15.368436 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:15.372429 (+ 3993us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:15.372676 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:04:15.373098 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36201,"thread_start_us":37850,"threads_started":1}
I20260929 03:04:15.377908 15964 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:15.378217 15964 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:15.379037 15964 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
W20260929 03:04:15.382323 15964 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:38231 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:04:15.382553 15964 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:15.382853 15964 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:15.384120 15964 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:15.384423 15964 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
W20260929 03:04:15.385687 15964 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:04:15.717397 15968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:15.717772 15968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:15.717928 15968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:15.725256 15968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:15.725368 15968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:15.725543 15968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:15.736034 15968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:40243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=46021
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:15.737476 15968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:15.738706 15968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:15.754204 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:15.755050 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:15.758666 15968 server_base.cc:1034] running on GCE node
W20260929 03:04:15.759016 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:15.759606 15968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:15.761031 15968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:15.762276 15968 hybrid_clock.cc:648] HybridClock initialized: now 1790651055762221 us; error 52 us; skew 500 ppm
I20260929 03:04:15.762578 15968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:15.769995 15968 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:15.772133 15968 webserver.cc:460] Webserver started at http://127.2.205.130:46021/ using document root <none> and password file <none>
I20260929 03:04:15.773146 15968 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:15.773294 15968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:15.832981 15968 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20260929 03:04:15.843370 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:15.845273 15968 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.010s sys 0.000s
I20260929 03:04:15.845468 15968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "adbe6c3527134a07ae53f5a6ea1707c7"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "50dec83418cc65de410c26f495518650"
server_key_iv: "73107310942a0e29ee395ab101d9b1ae"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:15.846894 15968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:15.869243 15968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:15.874600 15968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:15.874917 15968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:15.876178 15968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:15.878141 15968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:15.878252 15968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:15.878360 15968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:15.878451 15968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:15.956779 15968 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:40243
I20260929 03:04:15.956815 16107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:40243 every 8 connection(s)
I20260929 03:04:15.958175 15968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:15.959190 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 15968
I20260929 03:04:15.959932 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.205.148:40411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:41717
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:38231
--webserver_port=35373
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:15 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:15.992018 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:15.962354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:33607 (local address 127.2.205.190:38231)
0929 03:04:15.971373 (+ 9019us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:15.971380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:15.972268 (+ 888us) server_negotiation.cc:407] Connection header received
0929 03:04:15.975041 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:15.975048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:15.975139 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:15.975239 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:15.977001 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.977875 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.979458 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.980114 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.980239 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:15.984668 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:15.984675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:15.984680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:15.984721 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:15.988199 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:15.989157 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:15.989163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:15.989167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:15.989315 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:15.989864 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:15.989869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:15.989872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:15.990218 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:15.990345 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:15.991403 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0929 03:04:15.991631 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8877,"thread_start_us":8757,"threads_started":1}
I20260929 03:04:15.994570 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:15.961713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.130:33607)
0929 03:04:15.972014 (+ 10301us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:15.972070 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:15.974673 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:15.975625 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:15.975638 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:15.976182 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:15.976832 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:15.976853 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.978027 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.978031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:15.979307 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:15.979329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:15.980548 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:15.980556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:15.980869 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:15.981635 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:15.981665 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:15.984415 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:15.988363 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:15.988372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:15.988390 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:15.989016 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:15.989459 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:15.989464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:15.989468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:15.989722 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:15.990445 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:15.990453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:15.990966 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 03:04:15.992568 (+ 1602us) client_negotiation.cc:240] Negotiation successful
0929 03:04:15.993178 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9909,"thread_start_us":9688,"threads_started":1}
I20260929 03:04:15.996573 16108 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:15.996919 16108 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:15.997737 16108 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:15.999774 15767 ts_manager.cc:194] Registered new tserver with Master: adbe6c3527134a07ae53f5a6ea1707c7 (127.2.205.130:40243)
I20260929 03:04:16.001348 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:33607
Sep 29 03:04:16 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:16.385951 15964 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:16.386404 15964 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:16.387621 15767 ts_manager.cc:194] Registered new tserver with Master: 84dada6457e24f3cac5f4e3bedbb6991 (127.2.205.129:39603)
I20260929 03:04:16.388809 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:58217
W20260929 03:04:16.446365 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:16.446817 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:16.446983 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:16.459074 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:16.459231 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:16.459426 16113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:16.472642 16113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.205.148:40411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:41717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=35373
--tserver_master_addrs=127.2.205.190:38231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:16.474107 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:16.475440 16113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:16.490069 16119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:16.490271 16118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:16.490813 16121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:16.490875 16113 server_base.cc:1034] running on GCE node
I20260929 03:04:16.491665 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:16.497074 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:16.498354 16113 hybrid_clock.cc:648] HybridClock initialized: now 1790651056498283 us; error 81 us; skew 500 ppm
I20260929 03:04:16.498750 16113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:16 dist-test-slave-530n krb5kdc[14757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:16.505954 16113 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:16.508257 16113 webserver.cc:460] Webserver started at http://127.2.205.131:35373/ using document root <none> and password file <none>
I20260929 03:04:16.509313 16113 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:16.509478 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:04:16 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:16.550882 16113 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.010s sys 0.003s
I20260929 03:04:16.565445 16129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:16.570658 16113 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.004s
I20260929 03:04:16.570982 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "85a6574af0c345399d55a84274b7fafa"
format_stamp: "Formatted at 2026-09-29 03:04:13 on dist-test-slave-530n"
server_key: "adf88dc1082ac19aed0828826358ff4a"
server_key_iv: "f34f3ef5d85251f450b62a1bdccf123e"
server_key_version: "kuduclusterkey@0"
I20260929 03:04:16.572769 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:16.640769 16113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:16.646311 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:16.646708 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:16.648484 16113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:16.656801 16113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:16.656935 16113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 03:04:16.657052 16113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:16.657122 16113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:04:16 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for HTTP/127.2.205.148@KRBTEST.COM
I20260929 03:04:16.824225 16113 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41717
I20260929 03:04:16.824268 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41717 every 8 connection(s)
I20260929 03:04:16.825960 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:16.834102 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16113
Sep 29 03:04:16 dist-test-slave-530n krb5kdc[14757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:16.859841 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.836196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:36203 (local address 127.2.205.190:38231)
0929 03:04:16.836799 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.836807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.838744 (+ 1937us) server_negotiation.cc:407] Connection header received
0929 03:04:16.841588 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.841594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.841698 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.841772 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:16.844215 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.845396 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.847754 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.848388 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.848473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.852796 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:16.852804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:16.852809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:16.852849 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:16.856111 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.857050 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.857056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.857060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.857196 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.857706 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.857711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.857714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.858034 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:16.858162 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:16.859094 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 03:04:16.859459 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":437,"threads_started":1}
I20260929 03:04:16.860232 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.836528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.2.205.190:38231)
0929 03:04:16.837503 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.837507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.837522 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:04:16.837578 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.837581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.837657 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.837756 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:16.840885 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.841963 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.843989 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.844691 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.844906 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.848459 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:16.848464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:16.848469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:16.848508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:16.853617 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.857368 (+ 3751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.857372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.857375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.857500 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.857993 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.857998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.858002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.858300 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:16.859569 (+ 1269us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:16.859686 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:04:16.859903 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":794,"threads_started":1}
I20260929 03:04:16.862756 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.835530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:38231 (local address 127.2.205.131:36203)
0929 03:04:16.838570 (+ 3040us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:16.838629 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:16.841256 (+ 2627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:16.842427 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:16.842440 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:16.843096 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:16.843991 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.844019 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.845589 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.845594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.847576 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.847602 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.848860 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.848868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.849149 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.849781 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:16.849817 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:16.852472 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:16.856274 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:16.856285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:16.856320 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:16.856914 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:16.857326 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:16.857330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:16.857334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:16.857586 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:16.858240 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:16.858249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:16.858678 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 03:04:16.860471 (+ 1793us) client_negotiation.cc:240] Negotiation successful
0929 03:04:16.861020 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2705,"thread_start_us":2486,"threads_started":1}
I20260929 03:04:16.864399 16243 heartbeater.cc:347] Connected to a master server at 127.2.205.190:38231
I20260929 03:04:16.864701 16243 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:16.865545 16243 heartbeater.cc:511] Master 127.2.205.190:38231 requested a full tablet report, sending...
I20260929 03:04:16.867148 15767 ts_manager.cc:194] Registered new tserver with Master: 85a6574af0c345399d55a84274b7fafa (127.2.205.131:41717)
I20260929 03:04:16.868247 15767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:36203
I20260929 03:04:16.875550 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:04:16.925160 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.906656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.2.205.190:38231)
0929 03:04:16.906863 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.906870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.907446 (+ 576us) server_negotiation.cc:407] Connection header received
0929 03:04:16.907655 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.907661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.907751 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.907886 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:16.910072 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.911255 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.913359 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.914022 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.914175 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.917691 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:16.917701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:16.917706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:16.917759 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:16.921473 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.922535 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.922542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.922546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.922724 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:16.923262 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:16.923269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:16.923272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:16.923677 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:16.923931 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:16.924526 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 03:04:16.924817 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 03:04:16.932611 15767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:04:16.943037 15767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:04:16.993707 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.978776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:41717 (local address 127.0.0.1:43638)
0929 03:04:16.982988 (+ 4212us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:16.983008 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:16.983122 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:16.984818 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:16.984826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:16.984856 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:16.985226 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.985238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.987230 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.987236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.990189 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.990201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.992719 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.992725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.992957 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.993018 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 03:04:16.993146 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:04:16.993349 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4087,"thread_start_us":4423,"threads_started":1}
I20260929 03:04:16.994656 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.979150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.2.205.131:41717)
0929 03:04:16.983825 (+ 4675us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.983833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.983867 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:04:16.983969 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.983978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.984262 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.984478 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:16.985454 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.987065 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.990354 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.991934 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.992066 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.992316 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:16.994013 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0929 03:04:16.994195 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4509,"thread_start_us":4357,"threads_started":1}
I20260929 03:04:16.999035 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.965829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:39603 (local address 127.0.0.1:58680)
0929 03:04:16.977453 (+ 11624us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:16.977497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:16.977693 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:16.989614 (+ 11921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:16.989619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:16.989648 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:16.989961 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.989975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.994306 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.994311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.996165 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.996182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.998144 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.998151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.998332 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.998375 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:04:16.998453 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:04:16.998602 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11337,"mutex_wait_us":54,"thread_start_us":14082,"threads_started":1}
I20260929 03:04:16.999692 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.966202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40243 (local address 127.0.0.1:57172)
0929 03:04:16.979948 (+ 13746us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:16.979969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:16.980096 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:16.992087 (+ 11991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:16.992091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:16.992116 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:16.992457 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.992470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.994916 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.994927 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.997001 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:16.997017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.998900 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.998907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:16.999096 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:16.999132 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:04:16.999201 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:04:16.999334 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13609,"mutex_wait_us":58,"thread_start_us":11679,"threads_started":1}
I20260929 03:04:16.999210 16178 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9f3c887e21499e8279676c7d8458be (DEFAULT_TABLE table=test-table [id=142ad879567c4643b77921cf9e774bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:17.000736 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.966307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.2.205.130:40243)
0929 03:04:16.991557 (+ 25250us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.991566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.991591 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:04:16.991669 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.991675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.991879 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.992081 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:16.992677 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.994755 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.997192 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.998649 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.999936 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.000171 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.000248 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.000433 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25084,"thread_start_us":24965,"threads_started":1}
I20260929 03:04:17.000851 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:16.966220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.2.205.129:39603)
0929 03:04:16.988910 (+ 22690us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:16.988921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:16.988957 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:04:16.989072 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:16.989082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:16.989390 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:16.989631 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:16.992607 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.994143 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.996362 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:16.997895 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:16.999679 (+ 1784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.000266 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.000345 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.000520 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22438,"thread_start_us":26073,"threads_started":1}
I20260929 03:04:17.003358 16178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9f3c887e21499e8279676c7d8458be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:17.005223 16108 heartbeater.cc:503] Master 127.2.205.190:38231 was elected leader, sending a full tablet report...
I20260929 03:04:17.005435 16043 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9f3c887e21499e8279676c7d8458be (DEFAULT_TABLE table=test-table [id=142ad879567c4643b77921cf9e774bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:17.005414 15899 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9f3c887e21499e8279676c7d8458be (DEFAULT_TABLE table=test-table [id=142ad879567c4643b77921cf9e774bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:17.007031 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9f3c887e21499e8279676c7d8458be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:17.007054 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9f3c887e21499e8279676c7d8458be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:17.026553 16272 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Bootstrap starting.
I20260929 03:04:17.031369 16273 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Bootstrap starting.
I20260929 03:04:17.032456 16272 tablet_bootstrap.cc:654] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:17.033515 16271 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Bootstrap starting.
I20260929 03:04:17.037441 16271 tablet_bootstrap.cc:654] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:17.038048 16272 log.cc:824] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:17.038376 16273 tablet_bootstrap.cc:654] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:17.038815 16271 log.cc:824] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:17.039794 16273 log.cc:824] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:17.049079 16272 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: No bootstrap required, opened a new log
I20260929 03:04:17.049094 16271 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: No bootstrap required, opened a new log
I20260929 03:04:17.049446 16271 ts_tablet_manager.cc:1397] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260929 03:04:17.049546 16272 ts_tablet_manager.cc:1397] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.004s
I20260929 03:04:17.055833 16273 tablet_bootstrap.cc:492] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: No bootstrap required, opened a new log
I20260929 03:04:17.056177 16273 ts_tablet_manager.cc:1397] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.010s
I20260929 03:04:17.059614 16271 raft_consensus.cc:348] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.060197 16271 raft_consensus.cc:374] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:17.060361 16271 raft_consensus.cc:729] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84dada6457e24f3cac5f4e3bedbb6991, State: Initialized, Role: FOLLOWER
I20260929 03:04:17.061157 16271 consensus_queue.cc:260] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.061492 16272 raft_consensus.cc:348] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.061977 16272 raft_consensus.cc:374] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:17.062119 16272 raft_consensus.cc:729] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbe6c3527134a07ae53f5a6ea1707c7, State: Initialized, Role: FOLLOWER
I20260929 03:04:17.062814 16272 consensus_queue.cc:260] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.066330 16273 raft_consensus.cc:348] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.066833 16273 raft_consensus.cc:374] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:17.066972 16273 raft_consensus.cc:729] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a6574af0c345399d55a84274b7fafa, State: Initialized, Role: FOLLOWER
I20260929 03:04:17.067585 16273 consensus_queue.cc:260] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.070590 16271 ts_tablet_manager.cc:1428] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Time spent starting tablet: real 0.021s user 0.010s sys 0.009s
I20260929 03:04:17.071075 15964 heartbeater.cc:503] Master 127.2.205.190:38231 was elected leader, sending a full tablet report...
I20260929 03:04:17.071383 16272 ts_tablet_manager.cc:1428] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Time spent starting tablet: real 0.022s user 0.015s sys 0.003s
I20260929 03:04:17.075208 16273 ts_tablet_manager.cc:1428] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20260929 03:04:17.075716 16243 heartbeater.cc:503] Master 127.2.205.190:38231 was elected leader, sending a full tablet report...
W20260929 03:04:17.107702 16244 tablet.cc:2369] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:04:17.224521 16109 tablet.cc:2369] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:17.277606 16279 raft_consensus.cc:482] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:04:17.277977 16279 raft_consensus.cc:504] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.279711 16279 leader_election.cc:288] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84dada6457e24f3cac5f4e3bedbb6991 (127.2.205.129:39603), adbe6c3527134a07ae53f5a6ea1707c7 (127.2.205.130:40243)
I20260929 03:04:17.286243 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.279976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40243 (local address 127.2.205.131:48607)
0929 03:04:17.280260 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:17.280287 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:17.280446 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:17.280870 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:17.280875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:17.280908 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:17.281117 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:17.281126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.282341 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.282345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:17.284284 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:17.284301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.285551 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.285558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:17.285727 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.285770 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:04:17.285889 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:04:17.285986 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":25}
I20260929 03:04:17.286289 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.280102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48607 (local address 127.2.205.130:40243)
0929 03:04:17.280283 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.280290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.280420 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:04:17.280643 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.280648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.280722 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.280830 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:17.281245 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.282201 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.284471 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.285402 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.285530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.285585 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.285925 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.286052 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:04:17.286455 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.280293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48007 (local address 127.2.205.129:39603)
0929 03:04:17.280464 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.280470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.280870 (+ 400us) server_negotiation.cc:407] Connection header received
0929 03:04:17.281074 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.281078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.281147 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.281229 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:17.281670 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.282564 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.284804 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.285608 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.285761 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.285828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.286101 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.286226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:04:17.286556 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.280088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:39603 (local address 127.2.205.131:48007)
0929 03:04:17.280766 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:17.280780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:17.280953 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:17.281257 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:17.281261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:17.281282 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:17.281531 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:17.281538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.282702 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.282707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:17.284631 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:17.284640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.285784 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.285793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:17.285944 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.285975 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:04:17.286062 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:04:17.286177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":3,"spinlock_wait_cycles":22912,"thread_start_us":366,"threads_started":1}
I20260929 03:04:17.287458 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9f3c887e21499e8279676c7d8458be" candidate_uuid: "85a6574af0c345399d55a84274b7fafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" is_pre_election: true
I20260929 03:04:17.287552 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9f3c887e21499e8279676c7d8458be" candidate_uuid: "85a6574af0c345399d55a84274b7fafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dada6457e24f3cac5f4e3bedbb6991" is_pre_election: true
I20260929 03:04:17.287930 16063 raft_consensus.cc:2454] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a6574af0c345399d55a84274b7fafa in term 0.
I20260929 03:04:17.288018 15919 raft_consensus.cc:2454] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a6574af0c345399d55a84274b7fafa in term 0.
I20260929 03:04:17.288563 16131 leader_election.cc:302] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 84dada6457e24f3cac5f4e3bedbb6991, 85a6574af0c345399d55a84274b7fafa, adbe6c3527134a07ae53f5a6ea1707c7; no voters:
I20260929 03:04:17.288959 16279 raft_consensus.cc:2781] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:04:17.289105 16279 raft_consensus.cc:482] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:04:17.289198 16279 raft_consensus.cc:3037] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:17.291235 16279 raft_consensus.cc:504] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.291755 16279 leader_election.cc:288] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [CANDIDATE]: Term 1 election: Requested vote from peers 84dada6457e24f3cac5f4e3bedbb6991 (127.2.205.129:39603), adbe6c3527134a07ae53f5a6ea1707c7 (127.2.205.130:40243)
I20260929 03:04:17.292131 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9f3c887e21499e8279676c7d8458be" candidate_uuid: "85a6574af0c345399d55a84274b7fafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dada6457e24f3cac5f4e3bedbb6991"
I20260929 03:04:17.292161 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9f3c887e21499e8279676c7d8458be" candidate_uuid: "85a6574af0c345399d55a84274b7fafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbe6c3527134a07ae53f5a6ea1707c7"
I20260929 03:04:17.292292 15919 raft_consensus.cc:3037] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:17.292320 16063 raft_consensus.cc:3037] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:17.294428 15919 raft_consensus.cc:2454] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a6574af0c345399d55a84274b7fafa in term 1.
I20260929 03:04:17.294454 16063 raft_consensus.cc:2454] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a6574af0c345399d55a84274b7fafa in term 1.
I20260929 03:04:17.294790 16133 leader_election.cc:302] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84dada6457e24f3cac5f4e3bedbb6991, 85a6574af0c345399d55a84274b7fafa; no voters:
I20260929 03:04:17.295046 16279 raft_consensus.cc:2781] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:17.295475 16279 raft_consensus.cc:686] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 1 LEADER]: Becoming Leader. State: Replica: 85a6574af0c345399d55a84274b7fafa, State: Running, Role: LEADER
I20260929 03:04:17.295964 16279 consensus_queue.cc:237] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } }
I20260929 03:04:17.300379 15767 catalog_manager.cc:5495] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a6574af0c345399d55a84274b7fafa (127.2.205.131). New cstate: current_term: 1 leader_uuid: "85a6574af0c345399d55a84274b7fafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a6574af0c345399d55a84274b7fafa" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41717 } health_report { overall_health: HEALTHY } } }
W20260929 03:04:17.311923 15965 tablet.cc:2369] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:17.350780 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.345466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.2.205.131:41717)
0929 03:04:17.345674 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.345681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.345704 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:04:17.345812 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.345816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.345907 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.346055 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:04:17.346457 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.347171 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.348606 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.349051 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.349166 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.349697 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:04:17.350216 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:04:17.350351 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.350441 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.350525 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:04:17.358938 15919 raft_consensus.cc:1261] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 1 FOLLOWER]: Refusing update from remote peer 85a6574af0c345399d55a84274b7fafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:17.358948 16063 raft_consensus.cc:1261] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 1 FOLLOWER]: Refusing update from remote peer 85a6574af0c345399d55a84274b7fafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:17.359839 16281 consensus_queue.cc:1035] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "84dada6457e24f3cac5f4e3bedbb6991" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:17.360227 16279 consensus_queue.cc:1035] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbe6c3527134a07ae53f5a6ea1707c7" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:17.380645 16285 mvcc.cc:205] Tried to move back new op lower bound from 7334506730932477952 to 7334506730688114688. Current Snapshot: MvccSnapshot[applied={T|T < 7334506730932477952}]
I20260929 03:04:17.389391 15767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:17.389676 15767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:17.393240 15767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4de619118d84f198369ac0d24308c17: Sending DeleteTablet for 3 replicas of tablet 8e9f3c887e21499e8279676c7d8458be
I20260929 03:04:17.394426 16043 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9f3c887e21499e8279676c7d8458be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:57172
I20260929 03:04:17.394398 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9f3c887e21499e8279676c7d8458be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58680
I20260929 03:04:17.395102 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e9f3c887e21499e8279676c7d8458be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:17 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:43638
I20260929 03:04:17.395278 16296 tablet_replica.cc:330] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: stopping tablet replica
I20260929 03:04:17.395674 16296 raft_consensus.cc:2229] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:17.396175 16296 raft_consensus.cc:2258] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:17.397538 16297 tablet_replica.cc:330] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: stopping tablet replica
I20260929 03:04:17.398581 16297 raft_consensus.cc:2229] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:17.398983 16297 raft_consensus.cc:2258] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:17.399303 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15804
I20260929 03:04:17.401104 16297 ts_tablet_manager.cc:1905] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:17.401114 16296 ts_tablet_manager.cc:1905] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:17.408420 16297 ts_tablet_manager.cc:1918] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:17.408420 16296 ts_tablet_manager.cc:1918] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:17.408715 16296 log.cc:1196] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/8e9f3c887e21499e8279676c7d8458be
I20260929 03:04:17.408682 16297 log.cc:1196] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/8e9f3c887e21499e8279676c7d8458be
I20260929 03:04:17.409320 16297 ts_tablet_manager.cc:1939] T 8e9f3c887e21499e8279676c7d8458be P 84dada6457e24f3cac5f4e3bedbb6991: Deleting consensus metadata
I20260929 03:04:17.409322 16296 ts_tablet_manager.cc:1939] T 8e9f3c887e21499e8279676c7d8458be P adbe6c3527134a07ae53f5a6ea1707c7: Deleting consensus metadata
I20260929 03:04:17.411402 15754 catalog_manager.cc:4841] TS 84dada6457e24f3cac5f4e3bedbb6991 (127.2.205.129:39603): tablet 8e9f3c887e21499e8279676c7d8458be (table test-table [id=142ad879567c4643b77921cf9e774bba]) successfully deleted
I20260929 03:04:17.411357 15752 catalog_manager.cc:4841] TS adbe6c3527134a07ae53f5a6ea1707c7 (127.2.205.130:40243): tablet 8e9f3c887e21499e8279676c7d8458be (table test-table [id=142ad879567c4643b77921cf9e774bba]) successfully deleted
I20260929 03:04:17.412128 16298 tablet_replica.cc:330] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: stopping tablet replica
I20260929 03:04:17.412539 16298 raft_consensus.cc:2229] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:04:17.413184 16298 raft_consensus.cc:2258] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:17.415577 16298 ts_tablet_manager.cc:1905] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:17.423411 16298 ts_tablet_manager.cc:1918] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:17.423638 16298 log.cc:1196] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/8e9f3c887e21499e8279676c7d8458be
I20260929 03:04:17.424273 16298 ts_tablet_manager.cc:1939] T 8e9f3c887e21499e8279676c7d8458be P 85a6574af0c345399d55a84274b7fafa: Deleting consensus metadata
I20260929 03:04:17.426271 15752 catalog_manager.cc:4841] TS 85a6574af0c345399d55a84274b7fafa (127.2.205.131:41717): tablet 8e9f3c887e21499e8279676c7d8458be (table test-table [id=142ad879567c4643b77921cf9e774bba]) successfully deleted
I20260929 03:04:17.427592 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.400388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.2.205.129:39603)
0929 03:04:17.400552 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.400558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.414062 (+ 13504us) server_negotiation.cc:407] Connection header received
0929 03:04:17.414260 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.414264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.414366 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.414466 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:17.415899 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.416629 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.417988 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.418406 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.418520 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.421112 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:17.421127 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:17.421143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:17.421179 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:17.424183 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.425045 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.425053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.425056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.425198 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.425625 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.425631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.425633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.426109 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:17.426426 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.427018 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.427255 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:04:17.428270 15959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:17.518064 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15804
I20260929 03:04:17.532011 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15968
I20260929 03:04:17.545848 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.532863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.2.205.130:40243)
0929 03:04:17.533109 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.533115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.533134 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:17.533194 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.533198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.533268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.533354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:17.534706 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.535431 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.536727 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.537309 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.537428 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.540000 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:17.540015 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:17.540031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:17.540068 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:17.543034 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.543770 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.543780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.543782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.543936 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.544291 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.544297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.544300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.544655 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:17.544971 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.545286 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.545505 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 03:04:17.546566 16103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:17.644812 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15968
I20260929 03:04:17.658015 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16113
I20260929 03:04:17.671671 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:17.658805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.2.205.131:41717)
0929 03:04:17.658977 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:17.658984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:17.659056 (+ 72us) server_negotiation.cc:407] Connection header received
0929 03:04:17.659238 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:17.659244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:17.659314 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:17.659414 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:17.660668 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.661530 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.662891 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:17.663330 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:17.663456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:17.666022 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:17.666033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:17.666052 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:17.666083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:17.669026 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.669645 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.669652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.669658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.669784 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:17.670155 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:17.670158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:17.670160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:17.670524 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:17.670822 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:17.671111 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:04:17.671317 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:04:17.672411 16238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:17.793996 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16113
I20260929 03:04:17.809707 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 15734
I20260929 03:04:17.810917 15797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:17.922267 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 15734
I20260929 03:04:17.940696 2870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:04:18.276414 2870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:04:18.276609 2870 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:04:18.612151 2870 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:04:18.612 UTC [14787] LOG: received smart shutdown request
2026-09-29 03:04:18.615 UTC [14787] LOG: background worker "logical replication launcher" (PID 14796) exited with exit code 1
2026-09-29 03:04:18.615 UTC [14791] LOG: shutting down
2026-09-29 03:04:18.776 UTC [14787] LOG: database system is shut down
2026-09-29T03:04:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52112 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:04:18 dist-test-slave-530n krb5kdc[16312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:04:18 dist-test-slave-530n krb5kdc[16312](info): set up 2 sockets
Sep 29 03:04:18 dist-test-slave-530n krb5kdc[16312](info): commencing operation
krb5kdc: starting...
W20260929 03:04:19.901332 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:04:20 dist-test-slave-530n krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:04:20.117014 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35047
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:20.445096 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:20.445390 16325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:20.445524 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:20.452854 16325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:20.453018 16325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:20.453100 16325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:20.453181 16325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:20.453235 16325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:20.463127 16325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:20.464453 16325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:20.465766 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:20.477453 16330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:20.477430 16331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:20.477526 16333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:20.478313 16325 server_base.cc:1034] running on GCE node
I20260929 03:04:20.479246 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:20.480801 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:20.482043 16325 hybrid_clock.cc:648] HybridClock initialized: now 1790651060481986 us; error 51 us; skew 500 ppm
I20260929 03:04:20.482406 16325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:20 dist-test-slave-530n krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:20.485452 16325 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:20.486898 16325 webserver.cc:460] Webserver started at http://127.2.205.190:40835/ using document root <none> and password file <none>
I20260929 03:04:20.487624 16325 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:20.487748 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:20.488052 16325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:20.491281 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "dbb6cfdbf9a2412a820ec11e9bec472a"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "6cd4f10b9f6473e10c0b2f9831f3b3d1"
server_key_iv: "bbadbf46bf193214ade56d9ee8492c62"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.492139 16325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "dbb6cfdbf9a2412a820ec11e9bec472a"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "6cd4f10b9f6473e10c0b2f9831f3b3d1"
server_key_iv: "bbadbf46bf193214ade56d9ee8492c62"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.497548 16325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:04:20.501217 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:20.502802 16325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:04:20.502977 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "dbb6cfdbf9a2412a820ec11e9bec472a"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "6cd4f10b9f6473e10c0b2f9831f3b3d1"
server_key_iv: "bbadbf46bf193214ade56d9ee8492c62"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.503180 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:20.528136 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:20.532601 16325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:20.532914 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:20.551502 16325 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35047
I20260929 03:04:20.551548 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35047 every 8 connection(s)
I20260929 03:04:20.552903 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:20.556612 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:20.559754 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16325
I20260929 03:04:20.559964 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:04:20.560565 2870 external_mini_cluster.cc:1428] Setting key 46fedb21b54e59cb262105b21bd999fb
I20260929 03:04:20.563089 16392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:04:20 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.189@KRBTEST.COM" created.
I20260929 03:04:20.587462 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.566219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.2.205.190:35047)
0929 03:04:20.567874 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:20.567881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:20.567930 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:04:20.569575 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:20.569581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:20.569802 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:20.569977 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:20.571245 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:20.572421 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:20.573814 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:20.574596 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:20.574765 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:20.579443 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:20.579455 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:20.579473 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:20.579513 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:20.582905 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:20.583760 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:20.583769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:20.583773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:20.583955 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:20.584371 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:20.584376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:20.584379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:20.585063 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:20.585266 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:20.585890 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 03:04:20.586217 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"spinlock_wait_cycles":16640,"thread_start_us":1242,"threads_started":1}
I20260929 03:04:20.587504 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.566219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35047 (local address 127.0.0.1:42026)
0929 03:04:20.567088 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.567138 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:20.569184 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:20.570025 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:20.570031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:20.570707 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:20.571061 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:20.571080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:20.572570 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:20.572573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:20.573685 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:20.573697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:20.575024 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:20.575034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:20.575245 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:20.576068 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:20.576102 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:20.579105 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:20.583053 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:20.583066 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:20.583078 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:20.583616 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:20.584071 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:20.584076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:20.584079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:20.584271 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:20.585197 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:20.585209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:20.585467 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 03:04:20.585736 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 03:04:20.586415 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":137,"thread_start_us":396,"threads_started":1}
I20260929 03:04:20.590559 16392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:04:20.591513 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.591138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:38712)
0929 03:04:20.591294 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.591407 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:04:20.591729 16341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
I20260929 03:04:20.593232 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.592913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:38714)
0929 03:04:20.593047 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.593109 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:04:20.593403 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42105: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
I20260929 03:04:20.607750 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:42105
--webserver_interface=127.2.205.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:20.642702 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } attempt: 1
I20260929 03:04:20.643836 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.643387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:38722)
0929 03:04:20.643586 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.643704 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:04:20.644784 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.644408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:38736)
0929 03:04:20.644549 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.644617 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:04:20.644955 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42105: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
I20260929 03:04:20.713182 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } attempt: 2
I20260929 03:04:20.714396 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.713885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57900)
0929 03:04:20.714132 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.714230 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 03:04:20.715425 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.715048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57914)
0929 03:04:20.715207 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.715282 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:04:20.715605 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42105: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
I20260929 03:04:20.806897 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } attempt: 3
I20260929 03:04:20.808012 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.807531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57920)
0929 03:04:20.807733 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.807852 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 03:04:20.809021 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.808617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57926)
0929 03:04:20.808794 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.808877 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:04:20.809197 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42105: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
W20260929 03:04:20.928596 16399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:20.928980 16399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:20.929132 16399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:20.936647 16399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:20.936815 16399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:20.936894 16399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:20.936973 16399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:20.937044 16399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:20.947245 16399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:42105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.205.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:20.948577 16399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:20.949870 16399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:04:20.952458 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } attempt: 4
I20260929 03:04:20.953691 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.953147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57930)
0929 03:04:20.953462 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.953554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 03:04:20.954684 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:20.954282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57940)
0929 03:04:20.954423 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:20.954534 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:04:20.954916 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42105: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42105: connect: Connection refused (error 111)
W20260929 03:04:20.961042 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:20.961115 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:20.961069 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:20.961772 16399 server_base.cc:1034] running on GCE node
I20260929 03:04:20.963136 16399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:20.964519 16399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:20.965726 16399 hybrid_clock.cc:648] HybridClock initialized: now 1790651060965664 us; error 49 us; skew 500 ppm
I20260929 03:04:20.966081 16399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:20 dist-test-slave-530n krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:20.969118 16399 init.cc:377] Logged in from keytab as kudu/127.2.205.189@KRBTEST.COM (short username kudu)
I20260929 03:04:20.970636 16399 webserver.cc:460] Webserver started at http://127.2.205.189:33939/ using document root <none> and password file <none>
I20260929 03:04:20.971280 16399 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:20.971388 16399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:20.971657 16399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:20.974537 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data/instance:
uuid: "1f17f2ae94ad4e029fa4373c2568e259"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "c8a145f4730f31bae0df529d5f295de8"
server_key_iv: "441b58333c93b6531482dc8355029db4"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.975276 16399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal/instance:
uuid: "1f17f2ae94ad4e029fa4373c2568e259"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "c8a145f4730f31bae0df529d5f295de8"
server_key_iv: "441b58333c93b6531482dc8355029db4"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.980659 16399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:04:20.984648 16414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:20.986204 16399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 03:04:20.986392 16399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "1f17f2ae94ad4e029fa4373c2568e259"
format_stamp: "Formatted at 2026-09-29 03:04:20 on dist-test-slave-530n"
server_key: "c8a145f4730f31bae0df529d5f295de8"
server_key_iv: "441b58333c93b6531482dc8355029db4"
server_key_version: "encryptionkey@0"
I20260929 03:04:20.986603 16399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:21.006794 16399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:21.011274 16399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:21.011574 16399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:21.028824 16399 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.189:42105
I20260929 03:04:21.028870 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.189:42105 every 8 connection(s)
I20260929 03:04:21.030233 16399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
I20260929 03:04:21.034018 16466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:21.038693 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16399
I20260929 03:04:21.038868 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-1/wal/instance
I20260929 03:04:21.039418 2870 external_mini_cluster.cc:1428] Setting key e28b6fde59251b90caf578b7750377c2
I20260929 03:04:21.041558 16466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.188@KRBTEST.COM" created.
I20260929 03:04:21.061349 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.044926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.2.205.190:35047)
0929 03:04:21.045131 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.045137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.046260 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 03:04:21.048381 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.048388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.048470 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.048558 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.050096 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.050759 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.052245 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.052658 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.052784 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.056000 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.056005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.056009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.056042 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.058536 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.059217 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.059221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.059223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.059335 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.059770 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.059774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.059776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.060049 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.060198 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.060964 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.061137 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:04:21.062618 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.045279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35047 (local address 127.0.0.1:54940)
0929 03:04:21.046077 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.046131 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.048104 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.048832 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.048840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.049354 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.049928 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.049945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.050912 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.050916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.052110 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.052121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.052902 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.052910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.053098 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.053701 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.053723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.055789 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.058671 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.058679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.058694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.059101 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.059440 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.059445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.059449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.059646 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.060171 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.060179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.060533 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.060892 (+ 359us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.061352 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260929 03:04:21.064248 16466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:04:21.078599 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.064868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57954)
0929 03:04:21.065110 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.065128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.065222 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.066229 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.066233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.066563 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.066733 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.066743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.068038 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.068044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.069288 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.069302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.070066 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.070071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.070267 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.070768 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.070787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.073115 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.076256 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.076261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.076265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.076598 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.076986 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.076991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.076995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.077125 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.077698 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.077702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.077836 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.077952 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.078208 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 03:04:21.078693 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.065134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.2.205.189:42105)
0929 03:04:21.065779 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.065787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.065816 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:04:21.065874 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.065880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.066086 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.066221 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.066888 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.067901 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.069409 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.069922 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.070015 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.073291 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.073301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.073313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.073352 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.076145 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.076720 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.076729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.076733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.076892 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.077224 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.077228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.077232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.077582 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.077773 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.078105 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.078306 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":348,"threads_started":1}
I20260929 03:04:21.081136 16466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:04:21.082361 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.081918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59536)
0929 03:04:21.082094 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.082208 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 03:04:21.082686 16417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.084265 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.083940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59542)
0929 03:04:21.084076 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.084153 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:04:21.084475 16466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
I20260929 03:04:21.087437 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:46077
--webserver_interface=127.2.205.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:21.133795 16466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 1
I20260929 03:04:21.135023 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.134570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59548)
0929 03:04:21.134766 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.134872 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:04:21.136077 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.135729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59564)
0929 03:04:21.135886 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.135961 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:04:21.136302 16466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.204502 16466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 2
I20260929 03:04:21.205673 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.205227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59566)
0929 03:04:21.205414 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.205520 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:04:21.206673 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.206315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59580)
0929 03:04:21.206467 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.206555 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:04:21.206902 16466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.254202 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } attempt: 5
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:04:21.267073 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.254999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57960)
0929 03:04:21.255215 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.255237 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.255383 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.255735 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.255739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.256130 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.256310 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.256319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.257355 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.257360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.258223 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.258235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.258948 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.258953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.259128 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.259676 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.259694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.262020 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.264830 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.264835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.264839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.265195 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.265522 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.265527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.265531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.265663 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.266138 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.266143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.266297 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.266415 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.266717 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 03:04:21.267073 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.255052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.2.205.189:42105)
0929 03:04:21.255253 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.255258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.255320 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:04:21.255517 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.255521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.255601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.255685 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.256461 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.257225 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.258357 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.258809 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.258918 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.262200 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.262204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.262207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.262239 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.264697 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.265301 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.265306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.265308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.265423 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.265758 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.265762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.265765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.266029 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.266156 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.266535 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.266741 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 03:04:21.268112 16392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:04:21.269305 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.268938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59590)
0929 03:04:21.269095 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.269169 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:04:21.270270 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.269961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59596)
0929 03:04:21.270091 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.270170 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:04:21.270474 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.298105 16466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 3
I20260929 03:04:21.299345 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.298799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59604)
0929 03:04:21.299067 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.299176 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:04:21.300563 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.300100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59620)
0929 03:04:21.300343 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.300433 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260929 03:04:21.300812 16466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.321693 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 1
I20260929 03:04:21.322913 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.322397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59622)
0929 03:04:21.322622 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.322728 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 03:04:21.323899 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.323571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59632)
0929 03:04:21.323711 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.323785 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:04:21.324110 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.392299 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 2
I20260929 03:04:21.393512 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.393034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59636)
0929 03:04:21.393251 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.393342 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:04:21.394467 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.394154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59642)
0929 03:04:21.394283 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.394360 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:04:21.394677 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
W20260929 03:04:21.420039 16473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:21.420346 16473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:21.420445 16473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:21.427874 16473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:21.427982 16473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:21.428030 16473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:21.428085 16473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:21.428129 16473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:21.438333 16473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35047,127.2.205.189:42105,127.2.205.188:46077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.205.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:21.439836 16473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:21.441226 16473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:04:21.444015 16466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 4
I20260929 03:04:21.445070 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.444592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59644)
0929 03:04:21.444830 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.444913 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:04:21.446054 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.445755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59656)
0929 03:04:21.445885 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.445951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:04:21.446252 16466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
W20260929 03:04:21.453250 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:21.453217 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:21.453275 16481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:21.453920 16473 server_base.cc:1034] running on GCE node
I20260929 03:04:21.454612 16473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:21.455946 16473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:21.457157 16473 hybrid_clock.cc:648] HybridClock initialized: now 1790651061457108 us; error 37 us; skew 500 ppm
I20260929 03:04:21.457510 16473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:21.460597 16473 init.cc:377] Logged in from keytab as kudu/127.2.205.188@KRBTEST.COM (short username kudu)
I20260929 03:04:21.462215 16473 webserver.cc:460] Webserver started at http://127.2.205.188:38579/ using document root <none> and password file <none>
I20260929 03:04:21.462996 16473 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:21.463114 16473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:21.463421 16473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:21.466487 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data/instance:
uuid: "e37d0e92390c40e7b8ae459da4dac177"
format_stamp: "Formatted at 2026-09-29 03:04:21 on dist-test-slave-530n"
server_key: "7d7e1191a50cac768d75e8559022b049"
server_key_iv: "2ed220fd7ee0d8e25392085c2608f207"
server_key_version: "encryptionkey@0"
I20260929 03:04:21.467259 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal/instance:
uuid: "e37d0e92390c40e7b8ae459da4dac177"
format_stamp: "Formatted at 2026-09-29 03:04:21 on dist-test-slave-530n"
server_key: "7d7e1191a50cac768d75e8559022b049"
server_key_iv: "2ed220fd7ee0d8e25392085c2608f207"
server_key_version: "encryptionkey@0"
I20260929 03:04:21.472219 16473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:04:21.475660 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:21.477085 16473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:04:21.477223 16473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "e37d0e92390c40e7b8ae459da4dac177"
format_stamp: "Formatted at 2026-09-29 03:04:21 on dist-test-slave-530n"
server_key: "7d7e1191a50cac768d75e8559022b049"
server_key_iv: "2ed220fd7ee0d8e25392085c2608f207"
server_key_version: "encryptionkey@0"
I20260929 03:04:21.477383 16473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:21.490538 16473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:21.494292 16473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:21.494517 16473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:21.500842 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 3
I20260929 03:04:21.501961 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.501525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59672)
0929 03:04:21.501711 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.501802 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 03:04:21.503077 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.502605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59684)
0929 03:04:21.502826 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.502926 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 03:04:21.503306 16392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:46077: Network error: Client connection negotiation failed: client connection to 127.2.205.188:46077: connect: Connection refused (error 111)
I20260929 03:04:21.512117 16473 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.188:46077
I20260929 03:04:21.512140 16539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.188:46077 every 8 connection(s)
I20260929 03:04:21.513585 16473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
I20260929 03:04:21.517345 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:21.518915 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16473
I20260929 03:04:21.519119 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650859567927-2870-0/minicluster-data/master-2/wal/instance
I20260929 03:04:21.519652 2870 external_mini_cluster.cc:1428] Setting key 57543bbb8f26865ca75fc27fba089a63
I20260929 03:04:21.526625 16540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:04:21.543350 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.529138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.2.205.190:35047)
0929 03:04:21.529292 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.529297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.530106 (+ 809us) server_negotiation.cc:407] Connection header received
0929 03:04:21.530317 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.530323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.530394 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.530480 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.531637 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.532359 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.533668 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.534140 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.534234 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.537578 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.537586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.537591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.537637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.540236 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.541067 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.541073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.541077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.541223 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.541626 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.541632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.541635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.541907 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.542046 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.542744 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.542992 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:04:21.544461 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.521631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.2.205.188:46077)
0929 03:04:21.522439 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.522454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.522510 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:04:21.524075 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.524098 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.524528 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.524935 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.526369 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.527746 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.529464 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.530431 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.530605 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.538290 (+ 7685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.538301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.538319 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.538354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.541148 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.541835 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.541842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.541844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.541977 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.542367 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.542370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.542372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.542701 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.542994 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.543303 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.543532 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260929 03:04:21.544479 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.529160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35047 (local address 127.0.0.1:54948)
0929 03:04:21.529938 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.529993 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.530157 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.530573 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.530580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.531222 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.531486 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.531494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.532486 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.532489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.533541 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.533553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.534319 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.534323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.534454 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.535032 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.535053 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.537362 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.540405 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.540418 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.540432 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.540953 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.541325 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.541329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.541331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.541511 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.542030 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.542039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.542388 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.542628 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.543169 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":385,"threads_started":1}
I20260929 03:04:21.546066 16540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:21.561347 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.546732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42105 (local address 127.0.0.1:57964)
0929 03:04:21.546883 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.546903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.547015 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.547362 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.547367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.547848 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.548074 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.548085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.549482 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.549486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.550803 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.550820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.551811 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.551817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.552048 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.552743 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.552797 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.555273 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.558674 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.558680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.558684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.559231 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.559668 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.559673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.559677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.559847 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.560374 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.560380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.560543 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.560663 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.560989 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:04:21.561380 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.546936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.2.205.189:42105)
0929 03:04:21.547102 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.547106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.547122 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:04:21.547170 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.547173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.547231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.547308 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.548242 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.549339 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.550997 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.551635 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.551776 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.555535 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.555542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.555547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.555593 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.558529 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.559371 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.559377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.559381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.559558 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.559965 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.559970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.559974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.560257 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.560407 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.560798 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.561025 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:04:21.562557 16540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:04:21.564276 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.547608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.2.205.190:35047)
0929 03:04:21.547848 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.547853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.547873 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:04:21.547986 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.547992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.548066 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.548168 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.549768 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.550854 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.552409 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.553002 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.553132 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.557891 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.557898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.557903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.557947 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.561412 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.562301 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.562307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.562311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.562455 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.562865 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.562869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.562871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.563161 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.563441 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.563748 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.563965 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:04:21.576143 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.563242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59708)
0929 03:04:21.563518 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.563535 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.563618 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.563908 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.563913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.564354 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.564517 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.564525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.565542 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.565548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.566785 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.566800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.567483 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.567489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.567660 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.568149 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.568165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.570411 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.573171 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.573176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.573180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.573525 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.573889 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.573892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.573895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.574023 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.575126 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.575130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.575287 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.575469 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.575763 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260929 03:04:21.576180 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.563448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.2.205.188:46077)
0929 03:04:21.563605 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.563611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.563627 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:04:21.563709 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.563713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.563785 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.563871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.564638 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.565437 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.566926 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.567346 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.567442 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.570576 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.570581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.570584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.570610 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.573050 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.573627 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.573632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.573636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.573762 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.574129 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.574134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.574137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.574477 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.574623 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.575566 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.575895 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":46}
I20260929 03:04:21.580055 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.565824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.2.205.189:42105)
0929 03:04:21.565973 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.565977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.565991 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:04:21.566139 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.566143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.566193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.566261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.567582 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.568246 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.569847 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.570270 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.570360 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.574632 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.574637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.574640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.574670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.577602 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.578209 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.578215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.578217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.578346 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.578694 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.578697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.578699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.578954 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.579208 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.579479 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.579709 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:04:21.581287 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: Bootstrap starting.
I20260929 03:04:21.585168 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:04:21.585319 16540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:21.586652 16540 log.cc:824] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:21.590083 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: No bootstrap required, opened a new log
I20260929 03:04:21.600648 16540 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.602216 16540 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:21.602403 16540 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37d0e92390c40e7b8ae459da4dac177, State: Initialized, Role: FOLLOWER
I20260929 03:04:21.603389 16540 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.611213 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.591514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.2.205.190:35047)
0929 03:04:21.592406 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.592413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.592437 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:04:21.592515 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.592522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.592612 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.592727 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.595058 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.596152 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.598255 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.598909 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.599011 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.603281 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.603288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.603293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.603350 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.607309 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.608342 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.608350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.608355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.608571 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.609125 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.609131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.609135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.609552 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.609813 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.610502 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.610791 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747}
I20260929 03:04:21.612838 16553 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:21.613504 16553 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:21.614528 16540 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:21.618183 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.590786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.2.205.188:46077)
0929 03:04:21.591318 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.591324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.601419 (+ 10095us) server_negotiation.cc:407] Connection header received
0929 03:04:21.601577 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.601583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.601676 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.601786 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.604011 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.605125 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.607184 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.607927 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.608027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.611448 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.611456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.611461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.611506 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.615226 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.616055 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.616061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.616064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.616222 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.616615 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.616620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.616624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.617085 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.617350 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.617645 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.617850 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260929 03:04:21.623291 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.592181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.2.205.189:42105)
0929 03:04:21.592353 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.592358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.604448 (+ 12090us) server_negotiation.cc:407] Connection header received
0929 03:04:21.604728 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.604735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.604862 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.604940 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.607026 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.608136 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.610232 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.610874 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.610979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.615744 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.615752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.615757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.615813 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.619498 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.620272 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.620277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.620281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.620466 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.620808 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.620813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.620817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.621217 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.622472 (+ 1255us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.622618 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.622903 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 03:04:21.629108 16564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:04:21.629264 16564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:04:21.626051 16473 master_runner.cc:186] Error getting master registration for 127.2.205.190:35047: OK, instance_id { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" instance_seqno: 1790651060546305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:04:21.634397 16473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:04:21.680521 16392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 4
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:04:21.695724 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.681185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59732)
0929 03:04:21.681425 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.681446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.681575 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.682079 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.682083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.682622 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.682862 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.682876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.684316 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.684321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.685304 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.685318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.686299 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.686304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.686443 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.687080 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.687098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.689361 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.692882 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.692889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.692893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.693284 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.693683 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.693689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.693693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.693890 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.694648 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.694654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.694870 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.695019 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.695320 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 03:04:21.696826 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.681401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.2.205.188:46077)
0929 03:04:21.681591 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.681596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.681617 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:04:21.681678 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.681683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.681754 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.681865 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.683001 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.684168 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.685458 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.686138 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.686264 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.689574 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.689581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.689586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.689629 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.692683 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.693409 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.693415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.693419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.693572 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.694020 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.694026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.694030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.694409 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.694577 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.696097 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.696358 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:04:21.701454 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: Bootstrap starting.
I20260929 03:04:21.704830 16392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:21.707041 16392 log.cc:824] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:21.710690 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: No bootstrap required, opened a new log
I20260929 03:04:21.722059 16392 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.722710 16392 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:21.722860 16392 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb6cfdbf9a2412a820ec11e9bec472a, State: Initialized, Role: FOLLOWER
I20260929 03:04:21.723578 16392 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.724645 16567 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:21.725037 16567 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:21.726130 16392 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:04:21.735973 16578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:04:21.736217 16578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:04:21.745457 16466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } attempt: 5
W20260929 03:04:21.746733 16494 tablet.cc:2369] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:21.735378 16325 master_runner.cc:186] Error getting master registration for 127.2.205.189:42105: OK, instance_id { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" instance_seqno: 1790651061023807 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:04:21.749276 16325 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 03:04:21 dist-test-slave-530n krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:04:21.763222 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.746109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:46077 (local address 127.0.0.1:59734)
0929 03:04:21.746393 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:21.746414 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:21.746548 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:21.747075 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:21.747080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:21.747605 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:21.747803 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.747815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.749285 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.749292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.750694 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:21.750709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.751765 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.751771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:21.751977 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.752629 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:21.752653 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:21.755359 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:21.760528 (+ 5169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.760533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.760537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.761091 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.761471 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:21.761476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:21.761480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:21.761641 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:21.762226 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:21.762231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:21.762393 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 03:04:21.762546 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 03:04:21.762841 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 03:04:21.763274 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:21.746191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.2.205.188:46077)
0929 03:04:21.746424 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:21.746430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:21.746458 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:04:21.746717 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:21.746722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:21.746798 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:21.746891 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:21.747970 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.749111 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.750872 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:21.751607 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:21.751742 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:21.755597 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:21.755607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:21.755613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:21.755652 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:21.760389 (+ 4737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.761210 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.761216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.761220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.761360 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:21.761749 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:21.761754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:21.761757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:21.762103 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:21.762252 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:21.762647 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 03:04:21.762903 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 03:04:21.769425 16466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: Bootstrap starting.
I20260929 03:04:21.773367 16466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:21.774641 16466 log.cc:824] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:21.779403 16466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: No bootstrap required, opened a new log
W20260929 03:04:21.785990 16346 tablet.cc:2369] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:21.790920 16466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.791381 16466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:21.791554 16466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f17f2ae94ad4e029fa4373c2568e259, State: Initialized, Role: FOLLOWER
I20260929 03:04:21.792273 16466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.793332 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:21.793861 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:21.794723 16466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:21.818874 16399 master_runner.cc:263] No leader master found from master 1f17f2ae94ad4e029fa4373c2568e259
I20260929 03:04:21.819046 16399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:04:21.823619 16592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:04:21.823899 16592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:04:21.877853 16567 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:04:21.878166 16567 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.879796 16567 leader_election.cc:288] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f17f2ae94ad4e029fa4373c2568e259 (127.2.205.189:42105), e37d0e92390c40e7b8ae459da4dac177 (127.2.205.188:46077)
I20260929 03:04:21.881131 16515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37d0e92390c40e7b8ae459da4dac177" is_pre_election: true
I20260929 03:04:21.881789 16515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb6cfdbf9a2412a820ec11e9bec472a in term 0.
I20260929 03:04:21.882608 16343 leader_election.cc:302] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb6cfdbf9a2412a820ec11e9bec472a, e37d0e92390c40e7b8ae459da4dac177; no voters:
I20260929 03:04:21.883950 16441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f17f2ae94ad4e029fa4373c2568e259" is_pre_election: true
I20260929 03:04:21.884644 16441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb6cfdbf9a2412a820ec11e9bec472a in term 0.
I20260929 03:04:21.885869 16567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:04:21.886036 16567 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:04:21.886129 16567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:21.890046 16567 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.896024 16441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f17f2ae94ad4e029fa4373c2568e259"
I20260929 03:04:21.896277 16441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:21.899778 16441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb6cfdbf9a2412a820ec11e9bec472a in term 1.
I20260929 03:04:21.900411 16341 leader_election.cc:302] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f17f2ae94ad4e029fa4373c2568e259, dbb6cfdbf9a2412a820ec11e9bec472a; no voters:
I20260929 03:04:21.903707 16515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37d0e92390c40e7b8ae459da4dac177"
I20260929 03:04:21.903944 16515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:21.907907 16515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb6cfdbf9a2412a820ec11e9bec472a in term 1.
I20260929 03:04:21.908701 16567 leader_election.cc:288] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [CANDIDATE]: Term 1 election: Requested vote from peers 1f17f2ae94ad4e029fa4373c2568e259 (127.2.205.189:42105), e37d0e92390c40e7b8ae459da4dac177 (127.2.205.188:46077)
I20260929 03:04:21.909173 16567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:21.909605 16567 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [term 1 LEADER]: Becoming Leader. State: Replica: dbb6cfdbf9a2412a820ec11e9bec472a, State: Running, Role: LEADER
I20260929 03:04:21.910445 16567 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } }
I20260929 03:04:21.927435 16594 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb6cfdbf9a2412a820ec11e9bec472a. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:21.928081 16594 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:21.935207 16596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:21.937763 16596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:21.958072 16515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [term 1 FOLLOWER]: Refusing update from remote peer dbb6cfdbf9a2412a820ec11e9bec472a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:21.959391 16594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:21.966626 16441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [term 1 FOLLOWER]: Refusing update from remote peer dbb6cfdbf9a2412a820ec11e9bec472a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:21.967485 16594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:22.006590 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb6cfdbf9a2412a820ec11e9bec472a. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.006958 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:04:22.014321 16420 tablet.cc:2369] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:22.022750 16553 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb6cfdbf9a2412a820ec11e9bec472a. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.023170 16553 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:22.023205 16597 mvcc.cc:205] Tried to move back new op lower bound from 7334506749770285056 to 7334506749599363072. Current Snapshot: MvccSnapshot[applied={T|T < 7334506749770285056}]
I20260929 03:04:22.026575 16599 mvcc.cc:205] Tried to move back new op lower bound from 7334506749770285056 to 7334506749599363072. Current Snapshot: MvccSnapshot[applied={T|T < 7334506749770285056}]
I20260929 03:04:22.048919 16553 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.049199 16553 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:22.055141 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.055390 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:04:22.058894 16596 catalog_manager.cc:1347] Generated new cluster ID: 7f19a9e1620949d6bcf74a5cb2037181
I20260929 03:04:22.059007 16596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:22.060563 16567 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.060904 16567 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:22.061178 16567 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb6cfdbf9a2412a820ec11e9bec472a" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35047 } } peers { permanent_uuid: "1f17f2ae94ad4e029fa4373c2568e259" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42105 } } peers { permanent_uuid: "e37d0e92390c40e7b8ae459da4dac177" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 46077 } } }
I20260929 03:04:22.061411 16567 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:22.105976 16596 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:04:22.108203 16596 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:22.133301 16596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb6cfdbf9a2412a820ec11e9bec472a: Generated new TSK 0
I20260929 03:04:22.135067 16596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:04:22.631220 16564 catalog_manager.cc:1259] Loaded cluster ID: 7f19a9e1620949d6bcf74a5cb2037181
I20260929 03:04:22.631362 16564 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: loading cluster ID for follower catalog manager: success
I20260929 03:04:22.633980 16564 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: acquiring CA information for follower catalog manager: success
I20260929 03:04:22.635805 16564 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e37d0e92390c40e7b8ae459da4dac177: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:04:22.826001 16592 catalog_manager.cc:1259] Loaded cluster ID: 7f19a9e1620949d6bcf74a5cb2037181
I20260929 03:04:22.826134 16592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: loading cluster ID for follower catalog manager: success
I20260929 03:04:22.828686 16592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: acquiring CA information for follower catalog manager: success
I20260929 03:04:22.830520 16592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f17f2ae94ad4e029fa4373c2568e259: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:04:23.229059 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16325
I20260929 03:04:23.230348 16387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:23.349613 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16325
I20260929 03:04:23.366582 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16399
I20260929 03:04:23.367573 16461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:23.472374 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16399
I20260929 03:04:23.488911 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16473
I20260929 03:04:23.490042 16535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:23.596524 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16473
2026-09-29T03:04:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4711 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:04:23 dist-test-slave-530n krb5kdc[16614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:04:23 dist-test-slave-530n krb5kdc[16614](info): set up 2 sockets
Sep 29 03:04:23 dist-test-slave-530n krb5kdc[16614](info): commencing operation
krb5kdc: starting...
W20260929 03:04:24.609434 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:04:24 dist-test-slave-530n krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:04:24.828053 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37933
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:25.153175 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:25.153486 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:25.153620 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:25.160696 16627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:25.160846 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:25.160933 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:25.161011 16627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:25.161072 16627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:25.170621 16627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:37933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:25.171865 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:25.173166 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:25.184638 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:25.184623 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:25.184893 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:25.185495 16627 server_base.cc:1034] running on GCE node
I20260929 03:04:25.186396 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:25.187929 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:25.189137 16627 hybrid_clock.cc:648] HybridClock initialized: now 1790651065189088 us; error 36 us; skew 500 ppm
I20260929 03:04:25.189528 16627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:25 dist-test-slave-530n krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651065, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:25.192812 16627 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:25.194314 16627 webserver.cc:460] Webserver started at http://127.2.205.190:35115/ using document root <none> and password file <none>
I20260929 03:04:25.194998 16627 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:25.195099 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:25.195370 16627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:25.198606 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "ba62e73585b74338a952fd60364c84f6"
format_stamp: "Formatted at 2026-09-29 03:04:25 on dist-test-slave-530n"
server_key: "ddb128c5476dc67ae363a4847e9fa5a4"
server_key_iv: "7ff3d1e316bf39373d651031787701f2"
server_key_version: "encryptionkey@0"
I20260929 03:04:25.199394 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "ba62e73585b74338a952fd60364c84f6"
format_stamp: "Formatted at 2026-09-29 03:04:25 on dist-test-slave-530n"
server_key: "ddb128c5476dc67ae363a4847e9fa5a4"
server_key_iv: "7ff3d1e316bf39373d651031787701f2"
server_key_version: "encryptionkey@0"
I20260929 03:04:25.205511 16627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 03:04:25.209389 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:25.211159 16627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:04:25.211391 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "ba62e73585b74338a952fd60364c84f6"
format_stamp: "Formatted at 2026-09-29 03:04:25 on dist-test-slave-530n"
server_key: "ddb128c5476dc67ae363a4847e9fa5a4"
server_key_iv: "7ff3d1e316bf39373d651031787701f2"
server_key_version: "encryptionkey@0"
I20260929 03:04:25.211627 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:25.229501 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:25.234510 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:25.234858 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:25.252120 16627 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:37933
I20260929 03:04:25.252127 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:37933 every 8 connection(s)
I20260929 03:04:25.253492 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:25.257249 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:25.259860 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16627
I20260929 03:04:25.260092 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:04:25.260926 2870 external_mini_cluster.cc:1428] Setting key f79b02ef6d47ec50c9498eae54b58f8e
I20260929 03:04:25.269695 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6: Bootstrap starting.
I20260929 03:04:25.273443 16694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:25.274763 16694 log.cc:824] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:04:25 dist-test-slave-530n krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:25.278204 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6: No bootstrap required, opened a new log
I20260929 03:04:25.285683 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:25.262777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.2.205.190:37933)
0929 03:04:25.263555 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:25.263570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:25.263621 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:04:25.265639 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:25.265661 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:25.266062 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:25.266439 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:25.268189 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:25.269591 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:25.271413 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:25.272095 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:25.272219 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:25.277568 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:25.277582 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:25.277598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:25.277645 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:25.280979 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:25.281964 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:25.281971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:25.281978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:25.282171 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:25.282630 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:25.282634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:25.282637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:25.283299 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:25.283537 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:25.284088 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 03:04:25.284529 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":353,"threads_started":1}
I20260929 03:04:25.287423 16694 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } }
I20260929 03:04:25.287894 16694 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:25.288009 16694 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba62e73585b74338a952fd60364c84f6, State: Initialized, Role: FOLLOWER
I20260929 03:04:25.288615 16694 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } }
I20260929 03:04:25.288950 16694 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:25.289095 16694 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:25.289263 16694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:25.290099 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:04:25.291774 16694 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } }
I20260929 03:04:25.292379 16694 leader_election.cc:302] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba62e73585b74338a952fd60364c84f6; no voters:
I20260929 03:04:25.293118 16694 leader_election.cc:288] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:04:25.293221 16699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:25.293978 16699 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [term 1 LEADER]: Becoming Leader. State: Replica: ba62e73585b74338a952fd60364c84f6, State: Running, Role: LEADER
I20260929 03:04:25.294592 16699 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } }
I20260929 03:04:25.295403 16694 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:25.297469 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba62e73585b74338a952fd60364c84f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } } }
I20260929 03:04:25.297888 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:25.298609 16701 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba62e73585b74338a952fd60364c84f6. Latest consensus state: current_term: 1 leader_uuid: "ba62e73585b74338a952fd60364c84f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba62e73585b74338a952fd60364c84f6" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37933 } } }
I20260929 03:04:25.298940 16701 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:25.303100 16706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:25.307595 16706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:25.316828 16706 catalog_manager.cc:1347] Generated new cluster ID: 0c3c20e29cd0441baf652196d93d299c
I20260929 03:04:25.316967 16706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:25.332176 16706 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:04:25.334239 16706 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:25.344132 16706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba62e73585b74338a952fd60364c84f6: Generated new TSK 0
I20260929 03:04:25.345290 16706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:04:25.403249 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16627
I20260929 03:04:25.404647 16689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:25.502170 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16627
2026-09-29T03:04:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1903 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:04:25 dist-test-slave-530n krb5kdc[16722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:04:25 dist-test-slave-530n krb5kdc[16722](info): set up 2 sockets
Sep 29 03:04:25 dist-test-slave-530n krb5kdc[16722](info): commencing operation
krb5kdc: starting...
W20260929 03:04:26.533296 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:04:26 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:04:26.756122 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:27.080647 16735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:27.081010 16735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:27.081144 16735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:27.088587 16735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:27.088716 16735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:27.088804 16735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:27.088882 16735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:27.088948 16735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:27.099139 16735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:27.100589 16735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:27.101979 16735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:27.114058 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:27.113986 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:27.114095 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:27.114813 16735 server_base.cc:1034] running on GCE node
I20260929 03:04:27.115736 16735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:27.117280 16735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:27.118505 16735 hybrid_clock.cc:648] HybridClock initialized: now 1790651067118449 us; error 51 us; skew 500 ppm
I20260929 03:04:27.118880 16735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:27 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:27.129691 16735 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:27.131415 16735 webserver.cc:460] Webserver started at http://127.2.205.190:37513/ using document root <none> and password file <none>
I20260929 03:04:27.132117 16735 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:27.132220 16735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:27.132476 16735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:27.135576 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "16a1bfd8c99c40e6ae31d85ed84e3268"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "40ace61491958ed40adda6ce12123392"
server_key_iv: "bf58de8e919dd7a9683035e3e493deb7"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.136358 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "16a1bfd8c99c40e6ae31d85ed84e3268"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "40ace61491958ed40adda6ce12123392"
server_key_iv: "bf58de8e919dd7a9683035e3e493deb7"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.141588 16735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:04:27.145015 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:27.146481 16735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:04:27.146649 16735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "16a1bfd8c99c40e6ae31d85ed84e3268"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "40ace61491958ed40adda6ce12123392"
server_key_iv: "bf58de8e919dd7a9683035e3e493deb7"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.146839 16735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:27.166205 16735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:27.170770 16735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:27.171044 16735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:27.188526 16735 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35583
I20260929 03:04:27.188556 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35583 every 8 connection(s)
I20260929 03:04:27.190058 16735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:27.193939 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:27.197448 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16735
I20260929 03:04:27.197701 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:04:27.198536 2870 external_mini_cluster.cc:1428] Setting key 6a86cc3ebbbfa4fe20f78ce4383819b8
I20260929 03:04:27.206871 16802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap starting.
I20260929 03:04:27.210176 16802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:27.211321 16802 log.cc:824] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:04:27 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:27.214550 16802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: No bootstrap required, opened a new log
I20260929 03:04:27.221807 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:27.200767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.2.205.190:35583)
0929 03:04:27.201554 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:27.201568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:27.201615 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:04:27.203435 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:27.203462 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:27.203929 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:27.204351 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:27.205908 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.207252 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.208538 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.209312 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.209489 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:27.213727 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:27.213736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:27.213755 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:27.213798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:27.217189 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:27.217957 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:27.217966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:27.217973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:27.218157 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:27.218583 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:27.218587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:27.218589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:27.219234 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:27.219555 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:27.220031 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 03:04:27.220511 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":312,"threads_started":1}
I20260929 03:04:27.223173 16802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:27.223655 16802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:27.223766 16802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Initialized, Role: FOLLOWER
I20260929 03:04:27.224365 16802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:27.224592 16802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:27.224730 16802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:27.224973 16802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:27.227504 16802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:27.228060 16802 leader_election.cc:302] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a1bfd8c99c40e6ae31d85ed84e3268; no voters:
I20260929 03:04:27.228718 16802 leader_election.cc:288] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:04:27.228956 16807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:27.229610 16807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 LEADER]: Becoming Leader. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Running, Role: LEADER
I20260929 03:04:27.230290 16807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:27.231164 16802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:27.233561 16809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a1bfd8c99c40e6ae31d85ed84e3268. Latest consensus state: current_term: 1 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:27.233968 16809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:27.233707 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:27.234163 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:27.235667 16813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:27.250880 16813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:27.260142 16813 catalog_manager.cc:1347] Generated new cluster ID: 9f7533ce50ed46f7bae5e4bfe4074cb0
I20260929 03:04:27.260285 16813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:27.274415 16813 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:04:27.275725 16813 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:27.288401 16813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Generated new TSK 0
I20260929 03:04:27.289393 16813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:04:27.350734 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:27.686110 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:27.686432 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:27.686555 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:27.694061 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:27.694218 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:27.694397 16828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:27.704696 16828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:27.706168 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:27.707585 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:27.721827 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:27.721889 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:27.721906 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:27.722543 16828 server_base.cc:1034] running on GCE node
I20260929 03:04:27.723297 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:27.724361 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:27.725571 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790651067725503 us; error 62 us; skew 500 ppm
I20260929 03:04:27.725914 16828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:27 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:27.729187 16828 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:27.730723 16828 webserver.cc:460] Webserver started at http://127.2.205.129:37005/ using document root <none> and password file <none>
I20260929 03:04:27.731516 16828 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:27.731638 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:27.731932 16828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:27.735015 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "e6fc2bdf758e1d0654c810879b1a8173"
server_key_iv: "193cc4f997d56c93705e9f1e86e5e1ec"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.735757 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "e6fc2bdf758e1d0654c810879b1a8173"
server_key_iv: "193cc4f997d56c93705e9f1e86e5e1ec"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.741124 16828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:04:27.744661 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:27.746110 16828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:04:27.746265 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "e6fc2bdf758e1d0654c810879b1a8173"
server_key_iv: "193cc4f997d56c93705e9f1e86e5e1ec"
server_key_version: "encryptionkey@0"
I20260929 03:04:27.746450 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:27.758566 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:27.762524 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:27.762776 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:27.763652 16828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:27.764933 16828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:27.765007 16828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:27.765120 16828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:27.765169 16828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:27.798179 16828 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34165
I20260929 03:04:27.798179 16955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34165 every 8 connection(s)
I20260929 03:04:27.799712 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:27.807421 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16828
I20260929 03:04:27.807622 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:04:27.808277 2870 external_mini_cluster.cc:1428] Setting key ccd601f55fa4372c7ee23aadb130ab59
Sep 29 03:04:27 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:27.826771 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:27.803710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:43881 (local address 127.2.205.190:35583)
0929 03:04:27.804209 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:27.804214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:27.805159 (+ 945us) server_negotiation.cc:407] Connection header received
0929 03:04:27.807069 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:27.807075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:27.807168 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:27.807263 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:27.809149 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.810045 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.811811 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.812418 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.812547 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:27.816939 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:27.816948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:27.816953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:27.817005 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:27.821880 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:27.823218 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:27.823225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:27.823229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:27.823407 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:27.824175 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:27.824180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:27.824184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:27.824574 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:27.824721 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:27.826083 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0929 03:04:27.826354 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":273,"threads_started":1}
I20260929 03:04:27.829751 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:27.804261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:43881)
0929 03:04:27.804995 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:27.805037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:27.806761 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:27.807565 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:27.807577 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:27.808157 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:27.808914 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:27.808943 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.810234 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.810240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:27.811637 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:27.811651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:27.812624 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:27.812629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:27.812905 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:27.813635 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:27.813660 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:27.816566 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:27.822126 (+ 5560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:27.822138 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:27.822160 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:27.823036 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:27.823567 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:27.823572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:27.823575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:27.824026 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:27.824872 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:27.824881 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:27.825571 (+ 690us) client_negotiation.cc:769] Sending connection context
0929 03:04:27.827467 (+ 1896us) client_negotiation.cc:240] Negotiation successful
0929 03:04:27.828133 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":263,"threads_started":1}
I20260929 03:04:27.832230 16956 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:27.832711 16956 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:27.833758 16956 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:04:27.836663 16767 ts_manager.cc:194] Registered new tserver with Master: da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165)
I20260929 03:04:27.838943 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:43881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:04:27.869169 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:28.193284 16962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:28.193569 16962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:28.193701 16962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:28.201160 16962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:28.201277 16962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:28.201423 16962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:28.211380 16962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:28.212687 16962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:28.214054 16962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:28.227607 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:28.227564 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:28.227542 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:28.228349 16962 server_base.cc:1034] running on GCE node
I20260929 03:04:28.229058 16962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:28.230080 16962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:28.231312 16962 hybrid_clock.cc:648] HybridClock initialized: now 1790651068231251 us; error 43 us; skew 500 ppm
I20260929 03:04:28.231679 16962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:28 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651068, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:28.234874 16962 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:28.236330 16962 webserver.cc:460] Webserver started at http://127.2.205.130:34359/ using document root <none> and password file <none>
I20260929 03:04:28.237106 16962 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:28.237224 16962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:28.237524 16962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:28.240494 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "a66f372b1ec2de030cc0ac7c313db35d"
server_key_iv: "9bd46eff7e67fe32c52d1bb4d6b8ddce"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.241300 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "a66f372b1ec2de030cc0ac7c313db35d"
server_key_iv: "9bd46eff7e67fe32c52d1bb4d6b8ddce"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.246815 16962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:04:28.250748 16977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.252274 16962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:04:28.252457 16962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "a66f372b1ec2de030cc0ac7c313db35d"
server_key_iv: "9bd46eff7e67fe32c52d1bb4d6b8ddce"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.252658 16962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:28.272644 16962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:28.276876 16962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:28.277176 16962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:28.278203 16962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:28.279608 16962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:28.279690 16962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.279819 16962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:28.279871 16962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.311753 16962 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:40453
I20260929 03:04:28.311790 17089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:40453 every 8 connection(s)
I20260929 03:04:28.313153 16962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:28.319839 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 16962
I20260929 03:04:28.320035 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:04:28.320483 2870 external_mini_cluster.cc:1428] Setting key 8c451d0134e8f42926ea86561b179977
Sep 29 03:04:28 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651068, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:28.334520 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.315896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:41563 (local address 127.2.205.190:35583)
0929 03:04:28.316092 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.316097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.317431 (+ 1334us) server_negotiation.cc:407] Connection header received
0929 03:04:28.319449 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.319456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.319553 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.319646 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:28.321424 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.322409 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.324097 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.324539 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.324662 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.328340 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:28.328346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:28.328349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:28.328386 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:28.330974 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.332020 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.332024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.332026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.332168 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.332663 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.332667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.332669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.333013 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:28.333171 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.333928 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.334167 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:04:28.335567 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.316434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:41563)
0929 03:04:28.317238 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:28.317287 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:28.319130 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:28.319934 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:28.319944 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:28.320520 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:28.321236 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.321256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.322573 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.322578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.323922 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.323938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.324840 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.324851 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.325088 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.325758 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:28.325786 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:28.328062 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:28.331155 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:28.331166 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:28.331187 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:28.331875 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:28.332320 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:28.332326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:28.332330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:28.332540 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:28.333145 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:28.333155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:28.333545 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 03:04:28.333806 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 03:04:28.334395 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
I20260929 03:04:28.337231 17090 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:28.337563 17090 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:28.338346 17090 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
I20260929 03:04:28.339910 16767 ts_manager.cc:194] Registered new tserver with Master: d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453)
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:04:28.340822 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:41563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:04:28.368813 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:28.690393 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:28.690687 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:28.690811 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:28.698544 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:28.698665 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:28.698807 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:28.708858 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:28.710253 17095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:28.711663 17095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:28.725311 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:28.725282 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:28.725248 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:28.726100 17095 server_base.cc:1034] running on GCE node
I20260929 03:04:28.726703 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:28.727685 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:28.728837 17095 hybrid_clock.cc:648] HybridClock initialized: now 1790651068728775 us; error 82 us; skew 500 ppm
I20260929 03:04:28.729187 17095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:28 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651068, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:28.732478 17095 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:28.734129 17095 webserver.cc:460] Webserver started at http://127.2.205.131:34167/ using document root <none> and password file <none>
I20260929 03:04:28.734843 17095 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:28.734961 17095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:28.735262 17095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:28.738392 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "e546944ad8b34b08ace1c64a852f857d"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "5010616bb06d73efdf881ceeae4e8319"
server_key_iv: "ad03d745d63b6e94678a56e6a5062550"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.739158 17095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "e546944ad8b34b08ace1c64a852f857d"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "5010616bb06d73efdf881ceeae4e8319"
server_key_iv: "ad03d745d63b6e94678a56e6a5062550"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.744616 17095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:04:28.748299 17110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.750025 17095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:04:28.750190 17095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "e546944ad8b34b08ace1c64a852f857d"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "5010616bb06d73efdf881ceeae4e8319"
server_key_iv: "ad03d745d63b6e94678a56e6a5062550"
server_key_version: "encryptionkey@0"
I20260929 03:04:28.750388 17095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:28.762075 17095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:28.765640 17095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:28.765887 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:28.766742 17095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:28.767943 17095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:28.768015 17095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.768121 17095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:28.768172 17095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:28.797824 17095 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41017
I20260929 03:04:28.797844 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41017 every 8 connection(s)
I20260929 03:04:28.799214 17095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:28.799520 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17095
I20260929 03:04:28.799705 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:04:28.800269 2870 external_mini_cluster.cc:1428] Setting key 7a3a4b419a4759c5f5a236c48464a933
Sep 29 03:04:28 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651068, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:28.819538 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.802255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:36233 (local address 127.2.205.190:35583)
0929 03:04:28.802653 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.802659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.803828 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 03:04:28.805970 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.805976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.806060 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.806151 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:28.807637 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.808419 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.809989 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.810395 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.810514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.813958 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:28.813963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:28.813966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:28.814001 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:28.816618 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.817335 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.817339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.817341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.817471 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.817844 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.817848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.817850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.818152 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:28.818262 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.819098 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.819273 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260929 03:04:28.820588 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.802966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:36233)
0929 03:04:28.803686 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:28.803728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:28.805704 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:28.806385 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:28.806394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:28.806896 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:28.807470 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.807489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.808573 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.808577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.809841 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.809855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.810622 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.810629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.810855 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.811463 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:28.811487 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:28.813735 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:28.816735 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:28.816767 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:28.816787 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:28.817233 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:28.817553 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:28.817556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:28.817558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:28.817743 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:28.818291 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:28.818297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:28.818684 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 03:04:28.818973 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 03:04:28.819446 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":769,"threads_started":1}
I20260929 03:04:28.821969 17223 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:28.822275 17223 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:28.822997 17223 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
I20260929 03:04:28.824433 16767 ts_manager.cc:194] Registered new tserver with Master: e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017)
I20260929 03:04:28.825263 16767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:36233
I20260929 03:04:28.828879 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:04:28.844687 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.833203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.2.205.190:35583)
0929 03:04:28.833348 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.833353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.833790 (+ 437us) server_negotiation.cc:407] Connection header received
0929 03:04:28.833934 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.833937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.833992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.834067 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:28.835295 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.835997 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.837498 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.837886 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.837983 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.840135 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:28.840140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:28.840143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:28.840174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:28.842538 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.843182 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.843185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.843188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.843309 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:28.843575 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:28.843579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:28.843582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:28.843859 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:28.844068 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.844307 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.844467 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:04:28.846709 16956 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:28.848488 16766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:04:28.856112 16766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:04:28.890833 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.868117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.2.205.130:40453)
0929 03:04:28.881910 (+ 13793us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.881918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.881947 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:04:28.882041 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.882050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.882293 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.883110 (+ 817us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:28.883311 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.884891 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.887568 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.889595 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.889771 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.889995 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.890295 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.890484 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13623,"thread_start_us":13527,"threads_started":1}
I20260929 03:04:28.890977 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.867760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40453 (local address 127.0.0.1:51180)
0929 03:04:28.878974 (+ 11214us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:28.879016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:28.879202 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:28.882525 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:28.882533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:28.882565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:28.882931 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.882943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.885052 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.885058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.887427 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.887441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.889833 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.889840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.890035 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.890088 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:04:28.890249 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 03:04:28.890451 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10949,"mutex_wait_us":25,"thread_start_us":19000,"threads_started":1}
I20260929 03:04:28.891572 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.868147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.2.205.131:41017)
0929 03:04:28.881350 (+ 13203us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.881359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.882516 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 03:04:28.882620 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.882628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.882912 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.883099 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:28.884028 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.885668 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.888590 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.890026 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.890134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.890344 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.891089 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.891268 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12988,"thread_start_us":13205,"threads_started":1}
I20260929 03:04:28.892123 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.867883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:41017 (local address 127.0.0.1:58186)
0929 03:04:28.882242 (+ 14359us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:28.882261 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:28.882389 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:28.883382 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:28.883388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:28.883414 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:28.883811 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.883821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.886323 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.886329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.888412 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.888425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.890496 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.890502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.890682 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.890711 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:04:28.890804 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:04:28.890980 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14237,"mutex_wait_us":49,"thread_start_us":16953,"threads_started":1}
I20260929 03:04:28.895395 17158 tablet_service.cc:1467] Processing CreateTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e (DEFAULT_TABLE table=test-table [id=d660748feb8e41c995b2c8ae132a139f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:28.895411 17025 tablet_service.cc:1467] Processing CreateTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e (DEFAULT_TABLE table=test-table [id=d660748feb8e41c995b2c8ae132a139f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:28.896669 17158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0620d03fe0b4e6d992f19bf593bc45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:28.896718 17025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0620d03fe0b4e6d992f19bf593bc45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:28.898845 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.869413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:47678)
0929 03:04:28.887072 (+ 17659us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:28.887090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:28.887216 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:28.887830 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:28.887835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:28.887869 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:28.888245 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.888256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.890089 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.890093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.892088 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:28.892104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.897969 (+ 5865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.897978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:28.898173 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.898211 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:04:28.898308 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:04:28.898465 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17517,"mutex_wait_us":31,"thread_start_us":17060,"threads_started":1}
I20260929 03:04:28.899531 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:28.869565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.2.205.129:34165)
0929 03:04:28.886235 (+ 16670us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:28.886244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:28.887209 (+ 965us) server_negotiation.cc:407] Connection header received
0929 03:04:28.887353 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:28.887362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:28.887632 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:28.887823 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:28.888419 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.889965 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.892292 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:28.897313 (+ 5021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:28.897491 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:28.897738 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:28.898781 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0929 03:04:28.899205 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16497,"thread_start_us":16364,"threads_started":1}
I20260929 03:04:28.903761 16891 tablet_service.cc:1467] Processing CreateTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e (DEFAULT_TABLE table=test-table [id=d660748feb8e41c995b2c8ae132a139f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:28.905350 16891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0620d03fe0b4e6d992f19bf593bc45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:28.910928 17242 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Bootstrap starting.
I20260929 03:04:28.910930 17241 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Bootstrap starting.
I20260929 03:04:28.914705 17241 tablet_bootstrap.cc:654] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:28.914705 17242 tablet_bootstrap.cc:654] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:28.916020 17242 log.cc:824] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:28.916020 17241 log.cc:824] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:28.919977 17242 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: No bootstrap required, opened a new log
I20260929 03:04:28.920339 17242 ts_tablet_manager.cc:1397] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 03:04:28.923391 17241 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: No bootstrap required, opened a new log
I20260929 03:04:28.923682 17241 ts_tablet_manager.cc:1397] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260929 03:04:28.926666 17245 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Bootstrap starting.
I20260929 03:04:28.929554 17242 raft_consensus.cc:348] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.930150 17242 raft_consensus.cc:374] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:28.930258 17242 raft_consensus.cc:729] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e2ff50ff9d457182037a1a6fa4a10e, State: Initialized, Role: FOLLOWER
I20260929 03:04:28.930912 17245 tablet_bootstrap.cc:654] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:28.931026 17242 consensus_queue.cc:260] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.932340 17090 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:28.932355 17245 log.cc:824] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:28.932421 17241 raft_consensus.cc:348] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.932912 17241 raft_consensus.cc:374] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:28.933000 17241 raft_consensus.cc:729] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e546944ad8b34b08ace1c64a852f857d, State: Initialized, Role: FOLLOWER
I20260929 03:04:28.933142 17242 ts_tablet_manager.cc:1428] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 03:04:28.933658 17241 consensus_queue.cc:260] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.935432 17241 ts_tablet_manager.cc:1428] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260929 03:04:28.935830 17223 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:28.936196 17245 tablet_bootstrap.cc:492] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: No bootstrap required, opened a new log
I20260929 03:04:28.936487 17245 ts_tablet_manager.cc:1397] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260929 03:04:28.944403 17245 raft_consensus.cc:348] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.944792 17245 raft_consensus.cc:374] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:28.944890 17245 raft_consensus.cc:729] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2564ca5ad042a2bbb199ccb7c7cb3a, State: Initialized, Role: FOLLOWER
I20260929 03:04:28.945410 17245 consensus_queue.cc:260] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.947096 17245 ts_tablet_manager.cc:1428] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260929 03:04:28.997661 17248 raft_consensus.cc:482] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:04:28.997939 17248 raft_consensus.cc:504] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:28.999749 17248 leader_election.cc:288] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453), da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165)
I20260929 03:04:29.005177 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.000022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.2.205.131:43207)
0929 03:04:29.000321 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:29.000340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:29.000487 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:29.000805 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:29.000874 (+ 69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:29.000902 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:29.001084 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:29.001093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.002093 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.002097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:29.003382 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:29.003394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.004472 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.004477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:29.004641 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.004666 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:04:29.004763 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 03:04:29.004894 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 03:04:29.005175 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.000274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:43207 (local address 127.2.205.129:34165)
0929 03:04:29.000495 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.000500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.000519 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:29.000625 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.000630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.000688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.000791 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:29.001226 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.001964 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.003541 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.004316 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.004466 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.004541 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.004806 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.004940 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:04:29.005730 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.000136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:53877 (local address 127.2.205.130:40453)
0929 03:04:29.000324 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.000331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.000989 (+ 658us) server_negotiation.cc:407] Connection header received
0929 03:04:29.001148 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.001151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.001211 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.001281 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:29.001782 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.002473 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.004088 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.004876 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.005020 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.005089 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.005406 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.005536 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:04:29.006337 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.000117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40453 (local address 127.2.205.131:53877)
0929 03:04:29.000879 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:29.000895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:29.001019 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:29.001308 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:29.001312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:29.001334 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:29.001639 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:29.001648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.002615 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.002619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:29.003944 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:29.003954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.005073 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.005081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:29.005217 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.005244 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:04:29.005914 (+ 670us) client_negotiation.cc:240] Negotiation successful
0929 03:04:29.006041 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":435,"threads_started":1}
I20260929 03:04:29.006251 16911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0620d03fe0b4e6d992f19bf593bc45e" candidate_uuid: "e546944ad8b34b08ace1c64a852f857d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" is_pre_election: true
I20260929 03:04:29.006808 16911 raft_consensus.cc:2454] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e546944ad8b34b08ace1c64a852f857d in term 0.
I20260929 03:04:29.007267 17045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0620d03fe0b4e6d992f19bf593bc45e" candidate_uuid: "e546944ad8b34b08ace1c64a852f857d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" is_pre_election: true
I20260929 03:04:29.007464 17114 leader_election.cc:302] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da2564ca5ad042a2bbb199ccb7c7cb3a, e546944ad8b34b08ace1c64a852f857d; no voters:
I20260929 03:04:29.007794 17045 raft_consensus.cc:2454] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e546944ad8b34b08ace1c64a852f857d in term 0.
I20260929 03:04:29.007850 17248 raft_consensus.cc:2781] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:04:29.007972 17248 raft_consensus.cc:482] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:04:29.008060 17248 raft_consensus.cc:3037] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:29.010278 17248 raft_consensus.cc:504] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:29.010933 17248 leader_election.cc:288] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [CANDIDATE]: Term 1 election: Requested vote from peers d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453), da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165)
I20260929 03:04:29.011245 17045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0620d03fe0b4e6d992f19bf593bc45e" candidate_uuid: "e546944ad8b34b08ace1c64a852f857d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
I20260929 03:04:29.011454 17045 raft_consensus.cc:3037] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:29.011586 16911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0620d03fe0b4e6d992f19bf593bc45e" candidate_uuid: "e546944ad8b34b08ace1c64a852f857d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
I20260929 03:04:29.011780 16911 raft_consensus.cc:3037] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:29.014575 17045 raft_consensus.cc:2454] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e546944ad8b34b08ace1c64a852f857d in term 1.
I20260929 03:04:29.014710 16911 raft_consensus.cc:2454] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e546944ad8b34b08ace1c64a852f857d in term 1.
I20260929 03:04:29.014974 17113 leader_election.cc:302] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9e2ff50ff9d457182037a1a6fa4a10e, e546944ad8b34b08ace1c64a852f857d; no voters:
I20260929 03:04:29.015264 17248 raft_consensus.cc:2781] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:29.015627 17248 raft_consensus.cc:686] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 1 LEADER]: Becoming Leader. State: Replica: e546944ad8b34b08ace1c64a852f857d, State: Running, Role: LEADER
I20260929 03:04:29.016038 17248 consensus_queue.cc:237] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } }
I20260929 03:04:29.020955 16766 catalog_manager.cc:5495] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d reported cstate change: term changed from 0 to 1, leader changed from <none> to e546944ad8b34b08ace1c64a852f857d (127.2.205.131). New cstate: current_term: 1 leader_uuid: "e546944ad8b34b08ace1c64a852f857d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } health_report { overall_health: UNKNOWN } } }
W20260929 03:04:29.052134 16957 tablet.cc:2369] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:04:29.052349 17224 tablet.cc:2369] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:29.053120 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.047691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.2.205.131:41017)
0929 03:04:29.047874 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.047879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.047896 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:04:29.048004 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.048007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.048093 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.048213 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:04:29.048518 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.049256 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.050667 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.051092 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.051213 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.051642 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:04:29.052196 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:04:29.052340 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.052833 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.052920 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:04:29.060913 17045 raft_consensus.cc:1261] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Refusing update from remote peer e546944ad8b34b08ace1c64a852f857d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:29.060914 16911 raft_consensus.cc:1261] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Refusing update from remote peer e546944ad8b34b08ace1c64a852f857d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:29.061779 17248 consensus_queue.cc:1035] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:29.062045 17251 consensus_queue.cc:1035] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [LEADER]: Connected to new peer: Peer: permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:04:29.065085 17091 tablet.cc:2369] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:29.074220 17254 mvcc.cc:205] Tried to move back new op lower bound from 7334506778864394240 to 7334506778693349376. Current Snapshot: MvccSnapshot[applied={T|T < 7334506778864394240}]
I20260929 03:04:29.074592 17256 mvcc.cc:205] Tried to move back new op lower bound from 7334506778864394240 to 7334506778693349376. Current Snapshot: MvccSnapshot[applied={T|T < 7334506778864394240}]
I20260929 03:04:29.088552 16766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:29.088903 16766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:29.092890 16766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Sending DeleteTablet for 3 replicas of tablet c0620d03fe0b4e6d992f19bf593bc45e
I20260929 03:04:29.094167 17158 tablet_service.cc:1514] Processing DeleteTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:29 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:58186
I20260929 03:04:29.094177 17025 tablet_service.cc:1514] Processing DeleteTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:29 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:51180
I20260929 03:04:29.094336 16891 tablet_service.cc:1514] Processing DeleteTablet for tablet c0620d03fe0b4e6d992f19bf593bc45e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:29 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:47678
I20260929 03:04:29.099367 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16828
I20260929 03:04:29.109031 17264 tablet_replica.cc:330] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: stopping tablet replica
I20260929 03:04:29.109275 17266 tablet_replica.cc:330] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: stopping tablet replica
I20260929 03:04:29.109530 17265 tablet_replica.cc:330] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: stopping tablet replica
I20260929 03:04:29.109830 17266 raft_consensus.cc:2229] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:29.109839 17264 raft_consensus.cc:2229] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:04:29.109980 17265 raft_consensus.cc:2229] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:29.110455 17266 raft_consensus.cc:2258] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:29.110466 17265 raft_consensus.cc:2258] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:29.110630 17264 raft_consensus.cc:2258] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:29.112880 17266 ts_tablet_manager.cc:1905] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:29.112890 17264 ts_tablet_manager.cc:1905] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:29.120352 17264 ts_tablet_manager.cc:1918] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:29.120623 17264 log.cc:1196] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/c0620d03fe0b4e6d992f19bf593bc45e
I20260929 03:04:29.121099 17265 ts_tablet_manager.cc:1905] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:29.121289 17266 ts_tablet_manager.cc:1918] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:29.121492 17264 ts_tablet_manager.cc:1939] T c0620d03fe0b4e6d992f19bf593bc45e P e546944ad8b34b08ace1c64a852f857d: Deleting consensus metadata
I20260929 03:04:29.121512 17266 log.cc:1196] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/c0620d03fe0b4e6d992f19bf593bc45e
I20260929 03:04:29.122053 17266 ts_tablet_manager.cc:1939] T c0620d03fe0b4e6d992f19bf593bc45e P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting consensus metadata
I20260929 03:04:29.123701 16752 catalog_manager.cc:4841] TS e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017): tablet c0620d03fe0b4e6d992f19bf593bc45e (table test-table [id=d660748feb8e41c995b2c8ae132a139f]) successfully deleted
I20260929 03:04:29.123829 16754 catalog_manager.cc:4841] TS da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165): tablet c0620d03fe0b4e6d992f19bf593bc45e (table test-table [id=d660748feb8e41c995b2c8ae132a139f]) successfully deleted
Sep 29 03:04:29 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:04:29.128358 17265 ts_tablet_manager.cc:1918] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:29.128551 17265 log.cc:1196] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/c0620d03fe0b4e6d992f19bf593bc45e
I20260929 03:04:29.129134 17265 ts_tablet_manager.cc:1939] T c0620d03fe0b4e6d992f19bf593bc45e P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting consensus metadata
I20260929 03:04:29.131108 16753 catalog_manager.cc:4841] TS d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453): tablet c0620d03fe0b4e6d992f19bf593bc45e (table test-table [id=d660748feb8e41c995b2c8ae132a139f]) successfully deleted
I20260929 03:04:29.132368 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.100657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.2.205.129:34165)
0929 03:04:29.100928 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.100936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.104661 (+ 3725us) server_negotiation.cc:407] Connection header received
0929 03:04:29.105881 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.105889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.106008 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.106089 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:29.110942 (+ 4853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.112178 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.114763 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.115451 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.115561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.126020 (+ 10459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:29.126033 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:29.126049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:29.126090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:29.129042 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.129888 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.129896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.129899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.130044 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.130505 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.130511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.130514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.131170 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:29.131572 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.131822 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.132046 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:04:29.133081 16951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:29.223085 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16828
I20260929 03:04:29.244234 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16962
Sep 29 03:04:29 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:04:29.259318 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.245021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.2.205.130:40453)
0929 03:04:29.245222 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.245227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.245248 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:04:29.245391 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.245396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.245474 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.245567 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:29.246796 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.247516 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.248819 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.249260 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.249372 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.253457 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:29.253473 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:29.253489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:29.253530 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:29.256492 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.257330 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.257339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.257343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.257480 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.257839 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.257844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.257846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.258211 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:29.258499 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.258792 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.258997 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 03:04:29.259886 17085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:29.364450 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16962
I20260929 03:04:29.379428 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17095
Sep 29 03:04:29 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:04:29.396600 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:29.380226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.2.205.131:41017)
0929 03:04:29.380441 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:29.380447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:29.380468 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:04:29.380696 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:29.380700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:29.380801 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:29.380901 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:29.382292 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.383153 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.384495 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:29.385004 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:29.385132 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:29.389715 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:29.389732 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:29.389750 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:29.389797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:29.393449 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.394267 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.394277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.394279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.394491 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:29.394928 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:29.394932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:29.394934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:29.395329 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:29.395654 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:29.395974 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 03:04:29.396205 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:04:29.397290 17218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:29.520118 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17095
I20260929 03:04:29.536886 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 16735
I20260929 03:04:29.538021 16797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:29.648869 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 16735
I20260929 03:04:29.666448 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--webserver_interface=127.2.205.190
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:29.993474 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:29.993803 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:29.993938 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:30.001610 17272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:30.001744 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:30.001823 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:30.001906 17272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:30.001971 17272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:30.012382 17272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:30.013839 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:30.015304 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:30.038347 17278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.038375 17281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.038404 17279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:30.039330 17272 server_base.cc:1034] running on GCE node
I20260929 03:04:30.040290 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:30.042017 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:30.043252 17272 hybrid_clock.cc:648] HybridClock initialized: now 1790651070043190 us; error 43 us; skew 500 ppm
I20260929 03:04:30.043661 17272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:30 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651070, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:30.047186 17272 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:30.048870 17272 webserver.cc:460] Webserver started at http://127.2.205.190:37513/ using document root <none> and password file <none>
I20260929 03:04:30.049607 17272 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:30.049723 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:30.055485 17272 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:04:30.058444 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:30.059787 17272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:04:30.059958 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "16a1bfd8c99c40e6ae31d85ed84e3268"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "40ace61491958ed40adda6ce12123392"
server_key_iv: "bf58de8e919dd7a9683035e3e493deb7"
server_key_version: "encryptionkey@0"
I20260929 03:04:30.061018 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:30.072124 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:30.076288 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:30.076521 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:30.094789 17272 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35583
I20260929 03:04:30.094796 17339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35583 every 8 connection(s)
I20260929 03:04:30.096632 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:30.097184 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17272
I20260929 03:04:30.098172 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:34165
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=37005
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:30.104387 17340 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:04:30.109272 17340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap starting.
I20260929 03:04:30.138871 17340 log.cc:824] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:30.155148 17340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:04:30.155815 17340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap complete.
I20260929 03:04:30.168377 17340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:30.169281 17340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Initialized, Role: FOLLOWER
I20260929 03:04:30.169906 17340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:30.170137 17340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:30.170246 17340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:30.170406 17340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:04:30.174185 17340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:30.174670 17340 leader_election.cc:302] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a1bfd8c99c40e6ae31d85ed84e3268; no voters:
I20260929 03:04:30.175225 17340 leader_election.cc:288] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:04:30.175405 17344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:04:30.176431 17344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 LEADER]: Becoming Leader. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Running, Role: LEADER
I20260929 03:04:30.176995 17344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:30.177342 17340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:30.179432 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a1bfd8c99c40e6ae31d85ed84e3268. Latest consensus state: current_term: 2 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:30.179847 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:30.180429 17345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:30.180711 17345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:30.189100 17349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:30.198494 17349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:30.200258 17349 catalog_manager.cc:1259] Loaded cluster ID: 9f7533ce50ed46f7bae5e4bfe4074cb0
I20260929 03:04:30.200421 17349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:30.204071 17349 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:04:30.220216 17349 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:04:30.230513 17349 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:30.232501 17349 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Loaded TSK: 0
I20260929 03:04:30.234369 17349 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:04:30.464845 17342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:30.465174 17342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:30.465323 17342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:30.472615 17342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:30.472740 17342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:30.472952 17342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:30.483068 17342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:34165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=37005
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:30.484421 17342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:30.485843 17342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:30.501616 17367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.501591 17368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.502473 17370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:30.502856 17342 server_base.cc:1034] running on GCE node
I20260929 03:04:30.503710 17342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:30.505055 17342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:30.506301 17342 hybrid_clock.cc:648] HybridClock initialized: now 1790651070506246 us; error 42 us; skew 500 ppm
I20260929 03:04:30.506709 17342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:30 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651070, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:30.510768 17342 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:30.512580 17342 webserver.cc:460] Webserver started at http://127.2.205.129:37005/ using document root <none> and password file <none>
I20260929 03:04:30.513532 17342 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:30.513687 17342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:30.520211 17342 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 03:04:30.523425 17377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:30.524999 17342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:04:30.525149 17342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "e6fc2bdf758e1d0654c810879b1a8173"
server_key_iv: "193cc4f997d56c93705e9f1e86e5e1ec"
server_key_version: "encryptionkey@0"
I20260929 03:04:30.526404 17342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:30.554173 17342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:30.558552 17342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:30.558799 17342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:30.559756 17342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:30.561120 17342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:30.561220 17342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:30.561321 17342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:30.561388 17342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:30.591195 17342 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34165
I20260929 03:04:30.591261 17489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34165 every 8 connection(s)
I20260929 03:04:30.592971 17342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:30.594764 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17342
I20260929 03:04:30.595821 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:40453
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=34359
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:30 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651070, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:30.630220 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:30.597907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45823 (local address 127.2.205.190:35583)
0929 03:04:30.610345 (+ 12438us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:30.610361 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:30.610412 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:04:30.610782 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:30.610800 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:30.611221 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:30.611570 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:30.613775 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:30.615057 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:30.617252 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:30.617951 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:30.618089 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:30.622498 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:30.622506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:30.622524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:30.622563 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:30.625452 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:30.626386 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:30.626393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:30.626400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:30.626584 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:30.627051 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:30.627054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:30.627056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:30.627575 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:30.627727 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:30.628665 (+ 938us) server_negotiation.cc:299] Negotiation successful
0929 03:04:30.629111 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12206,"thread_start_us":12064,"threads_started":1}
I20260929 03:04:30.630436 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:30.596836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:45823)
0929 03:04:30.599352 (+ 2516us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:30.599404 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:30.602263 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:30.611835 (+ 9572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:30.611847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:30.612661 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:30.613487 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:30.613515 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:30.615232 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:30.615237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:30.617093 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:30.617112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:30.618199 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:30.618207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:30.618488 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:30.619235 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:30.619268 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:30.622174 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:30.625611 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:30.625622 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:30.625639 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:30.626251 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:30.626684 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:30.626688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:30.626693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:30.626957 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:30.627701 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:30.627710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:30.628145 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 03:04:30.628453 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 03:04:30.629083 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2156,"thread_start_us":1914,"threads_started":1}
I20260929 03:04:30.634186 17490 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:30.634603 17490 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:30.635598 17490 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
I20260929 03:04:30.638648 17305 ts_manager.cc:194] Registered new tserver with Master: da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165)
I20260929 03:04:30.640959 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:45823
W20260929 03:04:30.931746 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:30.932060 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:30.932157 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:30.939507 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:30.939628 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:30.939777 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:30.949775 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:40453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=34359
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:30.951090 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:30.952368 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:30.964866 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.964941 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:30.964972 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:30.965606 17494 server_base.cc:1034] running on GCE node
I20260929 03:04:30.966169 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:30.967032 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:30.968227 17494 hybrid_clock.cc:648] HybridClock initialized: now 1790651070968172 us; error 46 us; skew 500 ppm
I20260929 03:04:30.968513 17494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:30 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651070, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:30.971345 17494 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:30.972627 17494 webserver.cc:460] Webserver started at http://127.2.205.130:34359/ using document root <none> and password file <none>
I20260929 03:04:30.973289 17494 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:30.973397 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:30.978526 17494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 03:04:30.981307 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:30.982785 17494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:04:30.982986 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "a66f372b1ec2de030cc0ac7c313db35d"
server_key_iv: "9bd46eff7e67fe32c52d1bb4d6b8ddce"
server_key_version: "encryptionkey@0"
I20260929 03:04:30.983985 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:31.021179 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:31.025825 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:31.026103 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:31.027118 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:31.028473 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:31.028560 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:31.028645 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:31.028697 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:31.059468 17494 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:40453
I20260929 03:04:31.059536 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:40453 every 8 connection(s)
I20260929 03:04:31.060894 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:31.066960 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17494
I20260929 03:04:31.067525 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:41017
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=34167
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:04:31 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651070, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:31.079790 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.063358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:38703 (local address 127.2.205.190:35583)
0929 03:04:31.063555 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.063559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.064777 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 03:04:31.066626 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.066632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.066713 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.066798 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:31.068332 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.069064 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.070563 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.070975 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.071083 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.074332 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:31.074337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:31.074340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:31.074372 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:31.076741 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.077464 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.077470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.077473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.077610 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.078072 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.078078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.078082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.078384 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:31.078516 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.079212 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.079458 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:04:31.080735 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.063824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:38703)
0929 03:04:31.064599 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.064644 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.066367 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.067051 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.067059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.067564 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:31.068165 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.068184 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.069218 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.069222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.070422 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.070434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.071182 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.071189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.071410 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.072044 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:31.072072 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:31.074137 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:31.076905 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:31.076914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:31.076931 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:31.077354 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:31.077711 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:31.077717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:31.077721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:31.077970 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:31.078499 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:31.078506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:31.078862 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.079103 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.079603 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":271,"threads_started":1}
I20260929 03:04:31.082661 17622 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:31.083000 17622 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:31.083760 17622 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
I20260929 03:04:31.085287 17305 ts_manager.cc:194] Registered new tserver with Master: d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453)
I20260929 03:04:31.086117 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:38703
W20260929 03:04:31.394272 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:31.394550 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:31.394680 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:31.401760 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:31.401876 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:31.402040 17625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:31.412237 17625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:41017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=34167
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:31.413762 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:31.415148 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:31.428572 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:31.428602 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:31.428540 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:31.429314 17625 server_base.cc:1034] running on GCE node
I20260929 03:04:31.429963 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:31.431028 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:31.432252 17625 hybrid_clock.cc:648] HybridClock initialized: now 1790651071432193 us; error 54 us; skew 500 ppm
I20260929 03:04:31.432595 17625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:31 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:31.435694 17625 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:31.437328 17625 webserver.cc:460] Webserver started at http://127.2.205.131:34167/ using document root <none> and password file <none>
I20260929 03:04:31.438100 17625 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:31.438233 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:31.444175 17625 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:04:31.447110 17640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:31.448576 17625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:04:31.448740 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "e546944ad8b34b08ace1c64a852f857d"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "5010616bb06d73efdf881ceeae4e8319"
server_key_iv: "ad03d745d63b6e94678a56e6a5062550"
server_key_version: "encryptionkey@0"
I20260929 03:04:31.449873 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:31.504194 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:31.508265 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:31.508519 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:31.509541 17625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:31.510816 17625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:31.510926 17625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:31.511029 17625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:31.511097 17625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:31.540855 17625 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41017
I20260929 03:04:31.540889 17752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41017 every 8 connection(s)
I20260929 03:04:31.542492 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:31.550698 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17625
Sep 29 03:04:31 dist-test-slave-530n krb5kdc[16722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651071, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:31.562512 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.545018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:35311 (local address 127.2.205.190:35583)
0929 03:04:31.545207 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.545211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.546511 (+ 1300us) server_negotiation.cc:407] Connection header received
0929 03:04:31.548284 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.548289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.548370 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.548460 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:31.549966 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.550679 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.552569 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.553077 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.553196 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.556727 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:31.556731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:31.556735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:31.556794 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:31.559326 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.560013 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.560017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.560019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.560159 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.560586 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.560591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.560595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.561020 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:31.561182 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.561931 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.562145 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":69}
I20260929 03:04:31.563359 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.545621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:35311)
0929 03:04:31.546346 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.546385 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.548020 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.548694 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.548702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.549234 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:31.549796 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.549814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.550830 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.550835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.552433 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.552446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.553280 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.553288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.553559 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.554191 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:31.554215 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:31.556529 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:31.559455 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:31.559465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:31.559481 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:31.559900 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:31.560258 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:31.560263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:31.560267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:31.560468 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:31.561150 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:31.561155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:31.561506 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.561787 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.562182 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":278,"threads_started":1}
I20260929 03:04:31.564831 17753 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35583
I20260929 03:04:31.565150 17753 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:31.565630 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.551546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.2.205.190:35583)
0929 03:04:31.551979 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.551984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.552272 (+ 288us) server_negotiation.cc:407] Connection header received
0929 03:04:31.552428 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.552432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.552489 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.552574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:31.554147 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.555039 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.556626 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.557191 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.557306 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.559803 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:31.559808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:31.559812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:31.559841 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:31.562916 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.563565 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.563571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.563574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.563708 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.564130 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.564134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.564136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.564451 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:31.564623 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.565030 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.565240 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":321,"thread_start_us":199,"threads_started":1}
I20260929 03:04:31.565935 17753 heartbeater.cc:511] Master 127.2.205.190:35583 requested a full tablet report, sending...
I20260929 03:04:31.567601 17304 ts_manager.cc:194] Registered new tserver with Master: e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017)
I20260929 03:04:31.568535 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:35311
I20260929 03:04:31.580682 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:04:31.587903 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.581739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.2.205.190:35583)
0929 03:04:31.581914 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.581919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.582275 (+ 356us) server_negotiation.cc:407] Connection header received
0929 03:04:31.582468 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.582472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.582647 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.582796 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:04:31.583260 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.583973 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.585598 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.586049 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.586162 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.586661 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:04:31.587305 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:04:31.587433 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.587566 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.587648 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:04:31.589114 17304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:04:31.596544 17304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:04:31.626554 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.608674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48232)
0929 03:04:31.617397 (+ 8723us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.617420 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.617542 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.618527 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.618534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.618571 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:31.619484 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.619497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621371 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.621376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.623615 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.623629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.625448 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.625455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.625663 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.625716 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.625856 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.626072 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8588,"mutex_wait_us":48,"thread_start_us":9905,"threads_started":1}
I20260929 03:04:31.626736 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.609348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.2.205.130:40453)
0929 03:04:31.617045 (+ 7697us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.617056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.619016 (+ 1960us) server_negotiation.cc:407] Connection header received
0929 03:04:31.619164 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.619170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.619317 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.619481 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:31.620022 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621346 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.624251 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.625696 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.625827 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.626031 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.626258 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.626428 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7543,"thread_start_us":7387,"threads_started":1}
I20260929 03:04:31.627244 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.609148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40453 (local address 127.0.0.1:40078)
0929 03:04:31.618876 (+ 9728us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.618893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.619016 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.619500 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.619504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.619531 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:31.619861 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.619870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621503 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.621508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.624073 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.624090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.625914 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.625921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.626089 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.626120 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.626205 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.626343 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9587,"mutex_wait_us":46,"thread_start_us":7986,"threads_started":1}
I20260929 03:04:31.627354 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.608447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:41017 (local address 127.0.0.1:33286)
0929 03:04:31.616153 (+ 7706us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.616203 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.616404 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.618704 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.618712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.618735 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:31.619182 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.619198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621192 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.621197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.624298 (+ 3101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.624313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.626565 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.626573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.626729 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.626759 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.626820 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.626948 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7439,"mutex_wait_us":147,"spinlock_wait_cycles":3840,"thread_start_us":7275,"threads_started":1}
I20260929 03:04:31.627476 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.609213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.2.205.129:34165)
0929 03:04:31.617542 (+ 8329us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.617550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.617581 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:04:31.617689 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.617699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.618013 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.618220 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:31.619683 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621214 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.623785 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.625234 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.625389 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.625641 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.627009 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.627180 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8158,"thread_start_us":8026,"threads_started":1}
I20260929 03:04:31.628635 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.608729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.2.205.131:41017)
0929 03:04:31.618113 (+ 9384us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.618120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.618153 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:04:31.618242 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.618249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.618489 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.619062 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:31.619371 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.621030 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.624485 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.626315 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.627601 (+ 1286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.627835 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.627899 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.628057 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9214,"thread_start_us":9017,"threads_started":1}
I20260929 03:04:31.631898 17425 tablet_service.cc:1467] Processing CreateTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 (DEFAULT_TABLE table=test-table [id=8093471380454c4a823e1eab52603a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:31.631896 17688 tablet_service.cc:1467] Processing CreateTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 (DEFAULT_TABLE table=test-table [id=8093471380454c4a823e1eab52603a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:31.633505 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860ff67f5cdd4c83bfc3a48345249550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:31.633505 17425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860ff67f5cdd4c83bfc3a48345249550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:31.634259 17557 tablet_service.cc:1467] Processing CreateTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 (DEFAULT_TABLE table=test-table [id=8093471380454c4a823e1eab52603a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:31.635833 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 860ff67f5cdd4c83bfc3a48345249550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:31.645038 17490 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:31.649453 17769 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Bootstrap starting.
I20260929 03:04:31.651467 17770 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Bootstrap starting.
I20260929 03:04:31.653275 17769 tablet_bootstrap.cc:654] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:31.654618 17769 log.cc:824] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:31.654906 17770 tablet_bootstrap.cc:654] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:31.655730 17771 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Bootstrap starting.
I20260929 03:04:31.655905 17770 log.cc:824] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:31.658380 17769 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: No bootstrap required, opened a new log
I20260929 03:04:31.658581 17771 tablet_bootstrap.cc:654] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:31.658656 17769 ts_tablet_manager.cc:1397] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 03:04:31.659471 17771 log.cc:824] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:31.662211 17770 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: No bootstrap required, opened a new log
I20260929 03:04:31.662446 17770 ts_tablet_manager.cc:1397] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 03:04:31.662659 17771 tablet_bootstrap.cc:492] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: No bootstrap required, opened a new log
I20260929 03:04:31.662921 17771 ts_tablet_manager.cc:1397] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:04:31.668547 17769 raft_consensus.cc:348] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.668996 17769 raft_consensus.cc:374] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:31.669098 17769 raft_consensus.cc:729] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e2ff50ff9d457182037a1a6fa4a10e, State: Initialized, Role: FOLLOWER
I20260929 03:04:31.669680 17769 consensus_queue.cc:260] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.669827 17770 raft_consensus.cc:348] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.670212 17770 raft_consensus.cc:374] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:31.670315 17770 raft_consensus.cc:729] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2564ca5ad042a2bbb199ccb7c7cb3a, State: Initialized, Role: FOLLOWER
I20260929 03:04:31.670888 17622 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:31.670969 17770 consensus_queue.cc:260] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.671420 17769 ts_tablet_manager.cc:1428] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 03:04:31.672598 17771 raft_consensus.cc:348] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.673007 17771 raft_consensus.cc:374] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:31.673062 17770 ts_tablet_manager.cc:1428] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 03:04:31.673089 17771 raft_consensus.cc:729] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e546944ad8b34b08ace1c64a852f857d, State: Initialized, Role: FOLLOWER
I20260929 03:04:31.673630 17771 consensus_queue.cc:260] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.674620 17753 heartbeater.cc:503] Master 127.2.205.190:35583 was elected leader, sending a full tablet report...
I20260929 03:04:31.675165 17771 ts_tablet_manager.cc:1428] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260929 03:04:31.794162 17754 tablet.cc:2369] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:04:31.812841 17623 tablet.cc:2369] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:31.842020 17776 raft_consensus.cc:482] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:04:31.842286 17776 raft_consensus.cc:504] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.843986 17776 leader_election.cc:288] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017), d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453)
I20260929 03:04:31.850432 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.844432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:53847 (local address 127.2.205.130:40453)
0929 03:04:31.844648 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.844655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.845411 (+ 756us) server_negotiation.cc:407] Connection header received
0929 03:04:31.845593 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.845597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.845656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.845727 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:31.846341 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.847038 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.848833 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.849566 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.849712 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.849784 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.850068 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.850188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:04:31.850450 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.844522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:60169 (local address 127.2.205.131:41017)
0929 03:04:31.844723 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.844729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.845493 (+ 764us) server_negotiation.cc:407] Connection header received
0929 03:04:31.845647 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.845650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.845707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.845779 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:31.846399 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.847109 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.848832 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.849569 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.849712 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.849786 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.850105 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.850221 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:04:31.850577 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.844462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:40453 (local address 127.2.205.129:53847)
0929 03:04:31.845288 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.845305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.845461 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.845762 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.845767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.845791 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:31.846208 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.846217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.847154 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.847160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.848678 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.848690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.849770 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.849775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.849915 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.849954 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.850032 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.850156 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":56,"thread_start_us":445,"threads_started":1}
I20260929 03:04:31.850947 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.844591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:41017 (local address 127.2.205.129:60169)
0929 03:04:31.845382 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:31.845396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:31.845521 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:31.845801 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:31.845805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:31.845827 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:31.846239 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.846249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.847239 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.847245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.848689 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:31.848700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.849824 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.849831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:31.849966 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.849989 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:04:31.850065 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:04:31.850182 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":26,"thread_start_us":452,"threads_started":1}
I20260929 03:04:31.851641 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860ff67f5cdd4c83bfc3a48345249550" candidate_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" is_pre_election: true
I20260929 03:04:31.851833 17708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860ff67f5cdd4c83bfc3a48345249550" candidate_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e546944ad8b34b08ace1c64a852f857d" is_pre_election: true
I20260929 03:04:31.852087 17577 raft_consensus.cc:2454] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da2564ca5ad042a2bbb199ccb7c7cb3a in term 0.
I20260929 03:04:31.852312 17708 raft_consensus.cc:2454] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da2564ca5ad042a2bbb199ccb7c7cb3a in term 0.
I20260929 03:04:31.852733 17380 leader_election.cc:302] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9e2ff50ff9d457182037a1a6fa4a10e, da2564ca5ad042a2bbb199ccb7c7cb3a; no voters:
I20260929 03:04:31.853132 17776 raft_consensus.cc:2781] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:04:31.853252 17776 raft_consensus.cc:482] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:04:31.853367 17776 raft_consensus.cc:3037] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:31.856146 17776 raft_consensus.cc:504] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.856727 17776 leader_election.cc:288] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [CANDIDATE]: Term 1 election: Requested vote from peers e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017), d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453)
I20260929 03:04:31.857154 17708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860ff67f5cdd4c83bfc3a48345249550" candidate_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e546944ad8b34b08ace1c64a852f857d"
I20260929 03:04:31.857286 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "860ff67f5cdd4c83bfc3a48345249550" candidate_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
I20260929 03:04:31.857337 17708 raft_consensus.cc:3037] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:31.857503 17577 raft_consensus.cc:3037] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:31.859508 17708 raft_consensus.cc:2454] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da2564ca5ad042a2bbb199ccb7c7cb3a in term 1.
I20260929 03:04:31.859786 17577 raft_consensus.cc:2454] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da2564ca5ad042a2bbb199ccb7c7cb3a in term 1.
I20260929 03:04:31.859884 17379 leader_election.cc:302] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da2564ca5ad042a2bbb199ccb7c7cb3a, e546944ad8b34b08ace1c64a852f857d; no voters:
W20260929 03:04:31.860013 17491 tablet.cc:2369] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:04:31.860260 17776 raft_consensus.cc:2781] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:31.860731 17776 raft_consensus.cc:686] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 LEADER]: Becoming Leader. State: Replica: da2564ca5ad042a2bbb199ccb7c7cb3a, State: Running, Role: LEADER
I20260929 03:04:31.861289 17776 consensus_queue.cc:237] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } }
I20260929 03:04:31.865753 17305 catalog_manager.cc:5495] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a reported cstate change: term changed from 0 to 1, leader changed from <none> to da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129). New cstate: current_term: 1 leader_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 } health_report { overall_health: UNKNOWN } } }
I20260929 03:04:31.926061 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.920539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.2.205.129:34165)
0929 03:04:31.920779 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.920787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.920810 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:04:31.920913 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.920916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.920999 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.921158 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:04:31.921522 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.922293 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.923821 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.924239 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.924330 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.924943 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:04:31.925492 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:04:31.925613 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.925686 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.925764 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:04:31.935137 17577 raft_consensus.cc:1261] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Refusing update from remote peer da2564ca5ad042a2bbb199ccb7c7cb3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:31.935137 17708 raft_consensus.cc:1261] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Refusing update from remote peer da2564ca5ad042a2bbb199ccb7c7cb3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:31.936120 17780 consensus_queue.cc:1035] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e546944ad8b34b08ace1c64a852f857d" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:31.936480 17776 consensus_queue.cc:1035] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e2ff50ff9d457182037a1a6fa4a10e" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 40453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:31.961234 17305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:31.961524 17305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:31.964984 17305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Sending DeleteTablet for 3 replicas of tablet 860ff67f5cdd4c83bfc3a48345249550
I20260929 03:04:31.966120 17425 tablet_service.cc:1514] Processing DeleteTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:31 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:48232
I20260929 03:04:31.966135 17688 tablet_service.cc:1514] Processing DeleteTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:31 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:33286
I20260929 03:04:31.966538 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet 860ff67f5cdd4c83bfc3a48345249550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:31 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:40078
I20260929 03:04:31.967597 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17342
I20260929 03:04:31.978073 17795 tablet_replica.cc:330] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: stopping tablet replica
I20260929 03:04:31.978310 17793 tablet_replica.cc:330] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: stopping tablet replica
I20260929 03:04:31.978861 17793 raft_consensus.cc:2229] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:04:31.979512 17795 raft_consensus.cc:2229] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:31.979655 17793 raft_consensus.cc:2258] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:31.980875 17795 raft_consensus.cc:2258] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:31.981820 17794 tablet_replica.cc:330] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: stopping tablet replica
I20260929 03:04:31.982129 17793 ts_tablet_manager.cc:1905] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:31.982762 17794 raft_consensus.cc:2229] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:31.983434 17794 raft_consensus.cc:2258] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:31.984997 17795 ts_tablet_manager.cc:1905] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:31.986831 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:31.968385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.2.205.129:34165)
0929 03:04:31.968557 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:31.968563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:31.968668 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:04:31.968872 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:31.968878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:31.968971 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:31.969074 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:31.970717 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.972049 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.973882 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:31.974607 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:31.974716 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:31.978306 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:31.978322 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:31.978342 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:31.978403 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:31.982721 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.983960 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.983971 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.983974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.984169 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:31.984710 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:31.984715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:31.984719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:31.985224 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:31.985510 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:31.986180 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 03:04:31.986453 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:04:31.988286 17485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:31.990911 17794 ts_tablet_manager.cc:1905] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:31.993949 17795 ts_tablet_manager.cc:1918] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:31.994122 17795 log.cc:1196] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/860ff67f5cdd4c83bfc3a48345249550
I20260929 03:04:31.994657 17795 ts_tablet_manager.cc:1939] T 860ff67f5cdd4c83bfc3a48345249550 P d9e2ff50ff9d457182037a1a6fa4a10e: Deleting consensus metadata
I20260929 03:04:31.996646 17291 catalog_manager.cc:4841] TS d9e2ff50ff9d457182037a1a6fa4a10e (127.2.205.130:40453): tablet 860ff67f5cdd4c83bfc3a48345249550 (table test-table [id=8093471380454c4a823e1eab52603a8e]) successfully deleted
I20260929 03:04:31.997763 17794 ts_tablet_manager.cc:1918] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:31.997890 17794 log.cc:1196] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/860ff67f5cdd4c83bfc3a48345249550
I20260929 03:04:31.998342 17794 ts_tablet_manager.cc:1939] T 860ff67f5cdd4c83bfc3a48345249550 P e546944ad8b34b08ace1c64a852f857d: Deleting consensus metadata
I20260929 03:04:31.999787 17290 catalog_manager.cc:4841] TS e546944ad8b34b08ace1c64a852f857d (127.2.205.131:41017): tablet 860ff67f5cdd4c83bfc3a48345249550 (table test-table [id=8093471380454c4a823e1eab52603a8e]) successfully deleted
I20260929 03:04:32.086074 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17342
W20260929 03:04:32.102854 17292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:34165 (error 108)
I20260929 03:04:32.103143 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17494
I20260929 03:04:32.105973 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.105470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48270)
0929 03:04:32.105653 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.105786 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 03:04:32.106288 17292 catalog_manager.cc:4584] TS da2564ca5ad042a2bbb199ccb7c7cb3a (127.2.205.129:34165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 860ff67f5cdd4c83bfc3a48345249550: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
I20260929 03:04:32.118369 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.104116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.2.205.130:40453)
0929 03:04:32.104319 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:32.104326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:32.104419 (+ 93us) server_negotiation.cc:407] Connection header received
0929 03:04:32.104628 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:32.104634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:32.104726 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:32.104867 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:32.106854 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:32.107650 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:32.109207 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:32.109619 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:32.109738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:32.112442 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:32.112461 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:32.112479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:32.112519 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:32.115320 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:32.116105 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:32.116115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:32.116117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:32.116259 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:32.116669 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:32.116674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:32.116676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:32.117154 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:32.117449 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:32.117779 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 03:04:32.118021 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:04:32.119012 17617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:32.159796 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.159334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48280)
0929 03:04:32.159551 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.159652 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 03:04:32.160979 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.160580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48296)
0929 03:04:32.160721 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.160851 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:04:32.208724 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17494
I20260929 03:04:32.221740 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.221172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48300)
0929 03:04:32.221413 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.221531 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 03:04:32.222831 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.222439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48302)
0929 03:04:32.222576 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.222663 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:04:32.223278 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17625
I20260929 03:04:32.240317 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.224044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.2.205.131:41017)
0929 03:04:32.224225 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:32.224230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:32.224249 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:32.224444 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:32.224448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:32.224526 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:32.224613 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:32.225988 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:32.226728 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:32.228112 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:32.228516 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:32.228612 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:32.234729 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:32.234748 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:32.234766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:32.234807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:32.237640 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:32.238468 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:32.238476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:32.238479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:32.238633 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:32.239021 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:32.239025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:32.239027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:32.239379 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:32.239616 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:32.239875 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:04:32.240060 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:04:32.241055 17748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:32.303668 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.303038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48312)
0929 03:04:32.303293 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.303441 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 03:04:32.304883 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:32.304414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34165 (local address 127.0.0.1:48316)
0929 03:04:32.304614 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:32.304703 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 03:04:32.330016 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17625
I20260929 03:04:32.343559 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17272
I20260929 03:04:32.344794 17335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:32.460048 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17272
I20260929 03:04:32.477937 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--webserver_interface=127.2.205.190
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:32.808017 17801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:32.808332 17801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:32.808429 17801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:32.816020 17801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:32.816161 17801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:32.816206 17801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:32.816263 17801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:32.816308 17801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:32.826709 17801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:32.828223 17801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:32.829679 17801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:32.842319 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:32.842306 17809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:32.842371 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:32.843106 17801 server_base.cc:1034] running on GCE node
I20260929 03:04:32.843921 17801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:32.845544 17801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:32.846765 17801 hybrid_clock.cc:648] HybridClock initialized: now 1790651072846710 us; error 51 us; skew 500 ppm
I20260929 03:04:32.847137 17801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:32 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651072, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:32.850183 17801 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:32.851657 17801 webserver.cc:460] Webserver started at http://127.2.205.190:37513/ using document root <none> and password file <none>
I20260929 03:04:32.852317 17801 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:32.852432 17801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:32.857703 17801 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:04:32.860327 17816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:32.861563 17801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:04:32.861716 17801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "16a1bfd8c99c40e6ae31d85ed84e3268"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "40ace61491958ed40adda6ce12123392"
server_key_iv: "bf58de8e919dd7a9683035e3e493deb7"
server_key_version: "encryptionkey@0"
I20260929 03:04:32.862635 17801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:32.877866 17801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:32.882016 17801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:32.882264 17801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:32.900588 17801 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35583
I20260929 03:04:32.900609 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35583 every 8 connection(s)
I20260929 03:04:32.902173 17801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:32.909693 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17801
I20260929 03:04:32.910259 17868 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:04:32.910714 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:34165
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=37005
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:32.913758 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap starting.
I20260929 03:04:32.934579 17868 log.cc:824] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:32.951282 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:04:32.951840 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268: Bootstrap complete.
I20260929 03:04:32.960659 17868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:32.961594 17868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Initialized, Role: FOLLOWER
I20260929 03:04:32.962191 17868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:32.962419 17868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:32.962527 17868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:32.962693 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:04:32.966693 17868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:32.967168 17868 leader_election.cc:302] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a1bfd8c99c40e6ae31d85ed84e3268; no voters:
I20260929 03:04:32.967770 17868 leader_election.cc:288] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:04:32.967912 17872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:04:32.969885 17868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:32.974874 17872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [term 3 LEADER]: Becoming Leader. State: Replica: 16a1bfd8c99c40e6ae31d85ed84e3268, State: Running, Role: LEADER
I20260929 03:04:32.975489 17872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } }
I20260929 03:04:32.978473 17874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a1bfd8c99c40e6ae31d85ed84e3268. Latest consensus state: current_term: 3 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:32.978820 17874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:32.978444 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a1bfd8c99c40e6ae31d85ed84e3268" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35583 } } }
I20260929 03:04:32.979204 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a1bfd8c99c40e6ae31d85ed84e3268 [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:32.986896 17883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:32.993789 17883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:32.994980 17883 catalog_manager.cc:1259] Loaded cluster ID: 9f7533ce50ed46f7bae5e4bfe4074cb0
I20260929 03:04:32.995085 17883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 03:04:32.996627 17883 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7db2baa9ad google::LogMessage::Fail() at ??:0
@ 0x7f7db2bad88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7db2baa4de google::LogMessage::Flush() at ??:0
@ 0x7f7db2baba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7dc85dd512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7dc85da976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7dc8662610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7dc8662439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7db57f6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7db57f991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7db57f9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7db57d4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7db8aae6db start_thread at ??:0
@ 0x7f7db1a4461f clone at ??:0
W20260929 03:04:33.263806 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:33.264113 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:33.264241 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:33.273598 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:33.273748 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:33.273922 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:33.289014 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:34165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=37005
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:33.290333 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:33.291654 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:33.304891 17909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:33.304998 17912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:33.304955 17910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:33.305752 17870 server_base.cc:1034] running on GCE node
I20260929 03:04:33.306375 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:33.307382 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:33.308605 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790651073308557 us; error 41 us; skew 500 ppm
I20260929 03:04:33.308998 17870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:33 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:33.312067 17870 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:33.313593 17870 webserver.cc:460] Webserver started at http://127.2.205.129:37005/ using document root <none> and password file <none>
I20260929 03:04:33.314301 17870 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:33.314415 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:33.319939 17870 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:04:33.323006 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:33.324519 17870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:04:33.324705 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "da2564ca5ad042a2bbb199ccb7c7cb3a"
format_stamp: "Formatted at 2026-09-29 03:04:27 on dist-test-slave-530n"
server_key: "e6fc2bdf758e1d0654c810879b1a8173"
server_key_iv: "193cc4f997d56c93705e9f1e86e5e1ec"
server_key_version: "encryptionkey@0"
I20260929 03:04:33.325832 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:33.350534 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:33.354748 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:33.355017 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:33.356029 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:33.357860 17926 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 03:04:33.363163 17926 ts_tablet_manager.cc:1852] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 03:04:33.364110 17926 ts_tablet_manager.cc:1905] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:33.372659 17926 ts_tablet_manager.cc:1918] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:33.372848 17926 log.cc:1196] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/860ff67f5cdd4c83bfc3a48345249550
I20260929 03:04:33.373279 17926 ts_tablet_manager.cc:1939] T 860ff67f5cdd4c83bfc3a48345249550 P da2564ca5ad042a2bbb199ccb7c7cb3a: Deleting consensus metadata
I20260929 03:04:33.373623 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 03:04:33.373731 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260929 03:04:33.373845 17870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:33.373914 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:33.406570 17870 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34165
I20260929 03:04:33.406621 18032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34165 every 8 connection(s)
I20260929 03:04:33.408099 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:33.413576 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.411485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:41557)
0929 03:04:33.412289 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.412541 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":322,"threads_started":1}
I20260929 03:04:33.414229 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 17870
W20260929 03:04:33.414249 18033 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35583 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:35583: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
I20260929 03:04:33.414860 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:40453
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=34359
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:33.415198 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.414788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:34801)
0929 03:04:33.414975 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.415070 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:04:33.416124 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.415749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:47009)
0929 03:04:33.415883 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.415977 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:04:33.416394 18033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:04:33.745833 18036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:33.746155 18036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:33.746294 18036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:33.753612 18036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:33.753741 18036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:33.753893 18036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:33.763540 18036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:40453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=34359
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:33.764854 18036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:33.766139 18036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:33.783353 18041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:33.783444 18044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:33.783428 18042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:33.784329 18036 server_base.cc:1034] running on GCE node
I20260929 03:04:33.785043 18036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:33.786060 18036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:33.787264 18036 hybrid_clock.cc:648] HybridClock initialized: now 1790651073787211 us; error 47 us; skew 500 ppm
I20260929 03:04:33.787602 18036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:33 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:33.790735 18036 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:33.792295 18036 webserver.cc:460] Webserver started at http://127.2.205.130:34359/ using document root <none> and password file <none>
I20260929 03:04:33.793094 18036 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:33.793206 18036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:33.799099 18036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:04:33.802323 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:33.803927 18036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:04:33.804116 18036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "d9e2ff50ff9d457182037a1a6fa4a10e"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "a66f372b1ec2de030cc0ac7c313db35d"
server_key_iv: "9bd46eff7e67fe32c52d1bb4d6b8ddce"
server_key_version: "encryptionkey@0"
I20260929 03:04:33.805328 18036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:33.823518 18036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:33.828174 18036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:33.828459 18036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:33.829632 18036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:33.831027 18036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:33.831117 18036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:33.831221 18036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:33.831290 18036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:33.859993 18036 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:40453
I20260929 03:04:33.860049 18163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:40453 every 8 connection(s)
I20260929 03:04:33.861450 18036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:33.866117 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.864284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:39329)
0929 03:04:33.865032 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.865260 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":262,"threads_started":1}
W20260929 03:04:33.866797 18164 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35583 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:35583: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
I20260929 03:04:33.867532 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.867240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:56077)
0929 03:04:33.867374 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.867436 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:04:33.867583 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18036
I20260929 03:04:33.868350 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:41017
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:35583
--webserver_port=34167
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:04:33.868384 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:33.868093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:55817)
0929 03:04:33.868216 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:33.868285 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 03:04:33.868626 18164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:04:34.190992 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:34.191308 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:34.191401 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:34.198767 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:34.198894 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:34.199010 18167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:34.209134 18167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:41017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=34167
--tserver_master_addrs=127.2.205.190:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:34.210492 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:34.211833 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:34.225924 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:34.225841 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:34.225900 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:34.226653 18167 server_base.cc:1034] running on GCE node
I20260929 03:04:34.227308 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:34.228365 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:34.229568 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790651074229509 us; error 52 us; skew 500 ppm
I20260929 03:04:34.229931 18167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:34 dist-test-slave-530n krb5kdc[16722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:34.233178 18167 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:34.234666 18167 webserver.cc:460] Webserver started at http://127.2.205.131:34167/ using document root <none> and password file <none>
I20260929 03:04:34.235410 18167 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:34.235531 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:34.241179 18167 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:04:34.244086 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:34.245641 18167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:04:34.245821 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "e546944ad8b34b08ace1c64a852f857d"
format_stamp: "Formatted at 2026-09-29 03:04:28 on dist-test-slave-530n"
server_key: "5010616bb06d73efdf881ceeae4e8319"
server_key_iv: "ad03d745d63b6e94678a56e6a5062550"
server_key_version: "encryptionkey@0"
I20260929 03:04:34.246851 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:34.267392 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:34.271421 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:34.271663 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:34.272598 18167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:34.273912 18167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:34.274019 18167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:34.274130 18167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:34.274199 18167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:34.303893 18167 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:41017
I20260929 03:04:34.303980 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:41017 every 8 connection(s)
I20260929 03:04:34.305491 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:34.309629 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18167
I20260929 03:04:34.310583 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:34.308469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:33239)
0929 03:04:34.309295 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:34.309553 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":347,"threads_started":1}
W20260929 03:04:34.311256 18295 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:35583 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:35583: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
I20260929 03:04:34.312083 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:34.311765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:34181)
0929 03:04:34.311897 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:34.311972 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:04:34.312996 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:34.312584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:53063)
0929 03:04:34.312735 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:34.312874 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 03:04:34.313238 18295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:04:34.417959 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:34.417072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:54667)
0929 03:04:34.417590 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:34.417695 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
I20260929 03:04:34.869881 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:34.869151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:60483)
0929 03:04:34.869562 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:34.869669 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":192,"threads_started":1}
I20260929 03:04:35.314646 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:35.313794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:40803)
0929 03:04:35.314298 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:35.314408 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260929 03:04:35.419750 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:35.419026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:59597)
0929 03:04:35.419470 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:35.419581 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260929 03:04:35.871428 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:35.870735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:46411)
0929 03:04:35.871176 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:35.871256 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260929 03:04:36.316365 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:36.315605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:48255)
0929 03:04:36.316097 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:36.316190 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":222,"threads_started":1}
I20260929 03:04:36.421636 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:36.420797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:39325)
0929 03:04:36.421334 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:36.421436 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":260,"threads_started":1}
I20260929 03:04:36.873080 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:36.872286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:38235)
0929 03:04:36.872769 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:36.872868 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260929 03:04:37.318049 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:37.317336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:49191)
0929 03:04:37.317759 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:37.317847 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260929 03:04:37.423305 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:37.422568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:46959)
0929 03:04:37.423031 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:37.423119 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":217,"threads_started":1}
I20260929 03:04:37.874601 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:37.873919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:37737)
0929 03:04:37.874330 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:37.874411 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":202,"threads_started":1}
I20260929 03:04:38.319787 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:38.318993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:47047)
0929 03:04:38.319484 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:38.319588 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260929 03:04:38.425292 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:38.424343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:36761)
0929 03:04:38.424925 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:38.425097 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":264,"threads_started":1}
I20260929 03:04:38.876072 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:38.875417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:48781)
0929 03:04:38.875834 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:38.875911 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":207,"threads_started":1}
I20260929 03:04:39.321655 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:39.320819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:58209)
0929 03:04:39.321388 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:39.321480 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":302,"threads_started":1}
I20260929 03:04:39.426997 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:39.426222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:53881)
0929 03:04:39.426723 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:39.426802 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
I20260929 03:04:39.877655 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:39.876936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:42659)
0929 03:04:39.877398 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:39.877491 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260929 03:04:40.323629 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:40.322729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:58569)
0929 03:04:40.323296 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:40.323402 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":311,"threads_started":1}
I20260929 03:04:40.428918 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:40.427956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:38589)
0929 03:04:40.428541 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:40.428633 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":277,"threads_started":1}
I20260929 03:04:40.879426 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:40.878596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:43153)
0929 03:04:40.879087 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:40.879182 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":212,"threads_started":1}
I20260929 03:04:41.325465 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:41.324636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:45169)
0929 03:04:41.325152 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:41.325247 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":247,"threads_started":1}
I20260929 03:04:41.430521 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:41.429764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:59341)
0929 03:04:41.430222 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:41.430316 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":216,"threads_started":1}
I20260929 03:04:41.881202 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:41.880433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:37585)
0929 03:04:41.880934 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:41.881030 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":244,"threads_started":1}
I20260929 03:04:42.327198 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:42.326426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:50903)
0929 03:04:42.326902 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:42.326995 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260929 03:04:42.432144 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:42.431353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:46821)
0929 03:04:42.431855 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:42.431965 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260929 03:04:42.882949 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:42.882126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:51803)
0929 03:04:42.882664 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:42.882753 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":260,"threads_started":1}
I20260929 03:04:43.328856 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:43.328089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:39273)
0929 03:04:43.328548 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:43.328635 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":219,"threads_started":1}
I20260929 03:04:43.434021 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:43.433220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:32911)
0929 03:04:43.433727 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:43.433828 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
I20260929 03:04:43.884965 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:43.884102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:51665)
0929 03:04:43.884651 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:43.884779 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
I20260929 03:04:44.330549 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:44.329801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:40169)
0929 03:04:44.330281 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:44.330376 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":272,"threads_started":1}
I20260929 03:04:44.435864 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:44.435068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:42849)
0929 03:04:44.435569 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:44.435663 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260929 03:04:44.886837 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:44.886002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:56655)
0929 03:04:44.886524 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:44.886632 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260929 03:04:45.332350 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:45.331575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:35407)
0929 03:04:45.332060 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:45.332168 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":232,"threads_started":1}
I20260929 03:04:45.437753 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:45.436997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:60583)
0929 03:04:45.437471 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:45.437569 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260929 03:04:45.888880 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:45.888029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:57907)
0929 03:04:45.888570 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:45.888663 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
I20260929 03:04:46.334079 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:46.333379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:42561)
0929 03:04:46.333828 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:46.333918 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260929 03:04:46.439671 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:46.438786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:44363)
0929 03:04:46.439288 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:46.439431 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":225,"threads_started":1}
I20260929 03:04:46.890619 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:46.889850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:58297)
0929 03:04:46.890325 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:46.890421 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
I20260929 03:04:47.335755 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:47.334992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:49289)
0929 03:04:47.335448 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:47.335570 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260929 03:04:47.441264 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:47.440537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:35523)
0929 03:04:47.440993 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:47.441081 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260929 03:04:47.892254 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:47.891509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:43287)
0929 03:04:47.891982 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:47.892070 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260929 03:04:48.337330 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:48.336579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:47865)
0929 03:04:48.337073 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:48.337153 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260929 03:04:48.443274 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:48.442470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.129:37081)
0929 03:04:48.442943 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:48.443086 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260929 03:04:48.894187 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:48.893479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.130:50095)
0929 03:04:48.893904 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:48.894000 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260929 03:04:49.317047 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 17870
I20260929 03:04:49.335575 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:49.318290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.2.205.129:34165)
0929 03:04:49.318833 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:49.318847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:49.318885 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:04:49.320508 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:49.320521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:49.321027 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:49.321333 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:49.322705 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.323955 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.325327 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.326030 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.326166 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:49.328937 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:49.328946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:49.328965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:49.329012 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:49.331999 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.332831 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.332840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.332848 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.333074 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.333463 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.333468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.333472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.334055 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:49.334344 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:49.334824 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 03:04:49.335280 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260929 03:04:49.336995 18028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:49.341799 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:49.338425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35583 (local address 127.2.205.131:48361)
0929 03:04:49.341441 (+ 3016us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:49.341566 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:35583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2874,"thread_start_us":2749,"threads_started":1}
I20260929 03:04:49.421939 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 17870
I20260929 03:04:49.437371 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18036
I20260929 03:04:49.455279 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:49.438317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.2.205.130:40453)
0929 03:04:49.438846 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:49.438861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:49.438900 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:04:49.440511 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:49.440522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:49.441044 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:49.441357 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:49.442696 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.443975 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.445383 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.446120 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.446265 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:49.448896 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:49.448904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:49.448922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:49.448966 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:49.451887 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.452563 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.452570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.452576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.452740 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.453266 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.453270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.453272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.453966 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:49.454195 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:49.454589 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 03:04:49.454993 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260929 03:04:49.456809 18159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:49.540488 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18036
I20260929 03:04:49.555181 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18167
I20260929 03:04:49.572460 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:49.556082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.2.205.131:41017)
0929 03:04:49.556708 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:49.556721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:49.556786 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:04:49.558399 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:49.558408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:49.558831 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:49.559155 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:49.560591 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.561930 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.563187 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:49.563819 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:49.563991 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:49.566547 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:49.566554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:49.566569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:49.566607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:49.569324 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.570002 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.570016 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.570022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.570180 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:49.570543 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:49.570549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:49.570552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:49.571064 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:49.571345 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:49.571746 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 03:04:49.572160 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
I20260929 03:04:49.573935 18290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:49.656147 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18167
2026-09-29T03:04:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24153 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:04:49 dist-test-slave-530n krb5kdc[18352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:04:49 dist-test-slave-530n krb5kdc[18352](info): set up 2 sockets
Sep 29 03:04:49 dist-test-slave-530n krb5kdc[18352](info): commencing operation
krb5kdc: starting...
W20260929 03:04:50.661409 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:04:50 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:04:50.884308 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:51.210559 18365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:51.210924 18365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:51.211086 18365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:51.218609 18365 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:04:51.218750 18365 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:04:51.218808 18365 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:04:51.218880 18365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:04:51.218955 18365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:51.219058 18365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:51.219115 18365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:04:51.219190 18365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:04:51.229707 18365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:51.231477 18365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:51.233332 18365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:51.246241 18370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:51.246723 18371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:51.247059 18373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:51.248245 18365 server_base.cc:1034] running on GCE node
I20260929 03:04:51.249145 18365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:51.250553 18365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:51.251757 18365 hybrid_clock.cc:648] HybridClock initialized: now 1790651091251695 us; error 36 us; skew 500 ppm
I20260929 03:04:51.252072 18365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:51 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:51.258383 18365 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:04:51.259908 18365 webserver.cc:460] Webserver started at http://127.2.205.190:35375/ using document root <none> and password file <none>
I20260929 03:04:51.260625 18365 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:51.260721 18365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:51.261049 18365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:51.264215 18365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "dd4ee9e42b734b2db0434270cbf6944d"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "03a897fb63009286ac8fb53797be5fcd"
server_key_iv: "839dda0b333917934fba4ea8dbe06c86"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.265049 18365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "dd4ee9e42b734b2db0434270cbf6944d"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "03a897fb63009286ac8fb53797be5fcd"
server_key_iv: "839dda0b333917934fba4ea8dbe06c86"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.270150 18365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 03:04:51.273458 18380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:51.274735 18365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:04:51.274899 18365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "dd4ee9e42b734b2db0434270cbf6944d"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "03a897fb63009286ac8fb53797be5fcd"
server_key_iv: "839dda0b333917934fba4ea8dbe06c86"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.275110 18365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:51.309237 18365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:51.313701 18365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:51.313966 18365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:51.332136 18365 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:41047
I20260929 03:04:51.332129 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:41047 every 8 connection(s)
I20260929 03:04:51.333596 18365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:04:51.337031 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18365
I20260929 03:04:51.337232 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:04:51.337685 18432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:51.337881 2870 external_mini_cluster.cc:1428] Setting key 2982bdd1492ab8ac86a59f1dbd9475e7
I20260929 03:04:51.352345 18432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap starting.
Sep 29 03:04:51 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:51.356117 18432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:51.357511 18432 log.cc:824] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:51.361045 18432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: No bootstrap required, opened a new log
I20260929 03:04:51.362637 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:51.339992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.2.205.190:41047)
0929 03:04:51.340924 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:51.340940 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:51.340999 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:04:51.342842 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:51.342876 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:51.343497 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:51.344011 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:51.345454 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.347435 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.348798 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.349928 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.350130 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:51.354105 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:51.354116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:51.354135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:51.354185 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:51.357903 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:51.358809 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:51.358815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:51.358822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:51.359092 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:51.359463 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:51.359467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:51.359470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:51.360175 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:51.360448 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:51.360850 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:04:51.361311 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":380,"threads_started":1}
I20260929 03:04:51.368815 18432 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:04:51.369146 18432 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:51.369223 18432 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Initialized, Role: FOLLOWER
I20260929 03:04:51.369702 18432 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:04:51.369897 18432 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:04:51.370054 18432 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:04:51.370213 18432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:51.372272 18432 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:04:51.372731 18432 leader_election.cc:302] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd4ee9e42b734b2db0434270cbf6944d; no voters:
I20260929 03:04:51.373374 18432 leader_election.cc:288] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:04:51.373471 18437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:51.374137 18437 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 LEADER]: Becoming Leader. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Running, Role: LEADER
I20260929 03:04:51.374625 18437 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:04:51.375304 18432 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:04:51.377204 18439 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd4ee9e42b734b2db0434270cbf6944d. Latest consensus state: current_term: 1 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:04:51.377616 18439 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:51.378425 18438 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:04:51.378777 18438 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:04:51.380059 18443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:04:51.385712 18443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:04:51.395064 18443 catalog_manager.cc:1347] Generated new cluster ID: d0f6cd848f974963ba9ea9a08c1ad299
I20260929 03:04:51.395202 18443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:04:51.410566 18443 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:04:51.411976 18443 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:04:51.422385 18443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Generated new TSK 0
I20260929 03:04:51.423314 18443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:04:51.493377 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:51.831913 18458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:51.832242 18458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:51.832340 18458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:51.839834 18458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:51.839977 18458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:51.840155 18458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:04:51.851058 18458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:51.852361 18458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:51.853698 18458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:51.866505 18463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:51.866505 18464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:51.867125 18466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:51.867786 18458 server_base.cc:1034] running on GCE node
I20260929 03:04:51.868508 18458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:51.869772 18458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:51.873734 18458 hybrid_clock.cc:648] HybridClock initialized: now 1790651091873681 us; error 37 us; skew 500 ppm
I20260929 03:04:51.874131 18458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:51 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:51.878151 18458 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:04:51.879925 18458 webserver.cc:460] Webserver started at http://127.2.205.129:35425/ using document root <none> and password file <none>
I20260929 03:04:51.880867 18458 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:51.880997 18458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:51.881341 18458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:51.885531 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "309f1c09ef734c1080e50511298ac994"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "ae040536fb312e9dbf841d89fd7a8f4a"
server_key_iv: "b18d8fe84f2f1084f3f1bacf4ec413ab"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.886623 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "309f1c09ef734c1080e50511298ac994"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "ae040536fb312e9dbf841d89fd7a8f4a"
server_key_iv: "b18d8fe84f2f1084f3f1bacf4ec413ab"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.893433 18458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 03:04:51.897378 18473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:51.898959 18458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:04:51.899128 18458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "309f1c09ef734c1080e50511298ac994"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "ae040536fb312e9dbf841d89fd7a8f4a"
server_key_iv: "b18d8fe84f2f1084f3f1bacf4ec413ab"
server_key_version: "encryptionkey@0"
I20260929 03:04:51.899327 18458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:51.929592 18458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:51.934216 18458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:51.934518 18458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:51.935491 18458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:51.936884 18458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:51.936960 18458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:51.937072 18458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:51.937130 18458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:51.966869 18458 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34723
I20260929 03:04:51.966905 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34723 every 8 connection(s)
I20260929 03:04:51.968094 18458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:04:51.972083 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18458
I20260929 03:04:51.972282 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:04:51.972958 2870 external_mini_cluster.cc:1428] Setting key 842e2f1cd11b04b795ae37a3d750a560
Sep 29 03:04:51 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:51.988344 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:51.970499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:41935 (local address 127.2.205.190:41047)
0929 03:04:51.970965 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:51.970971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:51.971835 (+ 864us) server_negotiation.cc:407] Connection header received
0929 03:04:51.973758 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:51.973764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:51.973847 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:51.973962 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:51.975732 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.976482 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.978038 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.978432 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.978611 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:51.982254 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:51.982260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:51.982263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:51.982297 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:51.985214 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:51.986023 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:51.986028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:51.986031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:51.986181 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:51.986579 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:51.986583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:51.986586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:51.986925 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:51.987072 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:51.987736 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 03:04:51.987986 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":212,"threads_started":1}
I20260929 03:04:51.989290 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:51.971034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:41935)
0929 03:04:51.971689 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:51.971726 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:51.973453 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:51.974225 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:51.974236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:51.974940 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:51.975578 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:51.975597 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.976635 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.976639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:51.977903 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:51.977915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:51.978648 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:51.978656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:51.978942 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:51.979563 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:51.979585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:51.982007 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:51.985400 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:51.985409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:51.985424 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:51.985887 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:51.986274 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:51.986277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:51.986279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:51.986484 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:51.987027 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:51.987033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:51.987374 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 03:04:51.987601 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 03:04:51.988029 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":255,"threads_started":1}
I20260929 03:04:51.990863 18586 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:04:51.991180 18586 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:51.991940 18586 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:04:51.994371 18397 ts_manager.cc:194] Registered new tserver with Master: 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723)
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
I20260929 03:04:51.996454 18397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:41935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:04:52.025957 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:52.349462 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:52.349730 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:52.349822 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:52.357005 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:52.357128 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:52.357244 18592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:04:52.367041 18592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:52.368292 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:52.369620 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:52.382782 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:52.382705 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:52.382745 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:52.383829 18592 server_base.cc:1034] running on GCE node
I20260929 03:04:52.384431 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:52.385443 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:52.386667 18592 hybrid_clock.cc:648] HybridClock initialized: now 1790651092386609 us; error 41 us; skew 500 ppm
I20260929 03:04:52.387074 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:52 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:52.390225 18592 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:04:52.391723 18592 webserver.cc:460] Webserver started at http://127.2.205.130:39485/ using document root <none> and password file <none>
I20260929 03:04:52.392390 18592 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:52.392504 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:52.392844 18592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:52.395891 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "01319fab600141168ae2d62dbdf7b8cb"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "0a7d676deddda61e81fc46cee276d93e"
server_key_iv: "c6c466f5f2b80ad68e5c38fa1fc5abf0"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.396837 18592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "01319fab600141168ae2d62dbdf7b8cb"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "0a7d676deddda61e81fc46cee276d93e"
server_key_iv: "c6c466f5f2b80ad68e5c38fa1fc5abf0"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.402657 18592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:04:52.406472 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.408033 18592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:04:52.408202 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "01319fab600141168ae2d62dbdf7b8cb"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "0a7d676deddda61e81fc46cee276d93e"
server_key_iv: "c6c466f5f2b80ad68e5c38fa1fc5abf0"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.408398 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:52.431377 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:52.435698 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:52.435981 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:52.437016 18592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:52.438508 18592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:52.438598 18592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.438724 18592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:52.438776 18592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.470829 18592 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:35907
I20260929 03:04:52.470877 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:35907 every 8 connection(s)
I20260929 03:04:52.472069 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:04:52.479075 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18592
I20260929 03:04:52.479279 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:04:52.479907 2870 external_mini_cluster.cc:1428] Setting key 20574d47c7f78c34abd66ce4c85cf314
Sep 29 03:04:52 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:52.493007 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:52.474520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:51695 (local address 127.2.205.190:41047)
0929 03:04:52.474681 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:52.474686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:52.475771 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 03:04:52.477662 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:52.477666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:52.477753 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:52.477858 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:52.479834 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.480655 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.482538 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.482985 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.483090 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:52.487238 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:52.487243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:52.487246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:52.487282 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:52.489944 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:52.490666 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:52.490671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:52.490675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:52.490808 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:52.491265 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:52.491269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:52.491271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:52.491567 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:52.491684 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:52.492504 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 03:04:52.492686 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:04:52.494379 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:52.474903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:51695)
0929 03:04:52.475607 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:52.475642 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:52.477370 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:52.478110 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:52.478120 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:52.478724 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:52.479605 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:52.479642 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.480852 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.480857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:52.482388 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:52.482404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.483217 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.483224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:52.483519 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:52.484233 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:52.484262 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:52.486963 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:52.490078 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:52.490089 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:52.490109 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:52.490540 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:52.490911 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:52.490916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:52.490919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:52.491152 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:52.491680 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:52.491688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:52.492123 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 03:04:52.492416 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 03:04:52.492996 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260929 03:04:52.495832 18720 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:04:52.496165 18720 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:52.497037 18720 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:04:52.498546 18397 ts_manager.cc:194] Registered new tserver with Master: 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907)
I20260929 03:04:52.499362 18397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:51695
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:04:52.530500 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:04:52.850014 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:04:52.850292 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:04:52.850431 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:04:52.857453 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:04:52.857568 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:04:52.857715 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:04:52.867304 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:04:52.868492 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:04:52.869743 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:04:52.881919 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:52.881888 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:04:52.881969 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:04:52.882752 18725 server_base.cc:1034] running on GCE node
I20260929 03:04:52.883435 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:04:52.884503 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:04:52.885722 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790651092885665 us; error 52 us; skew 500 ppm
I20260929 03:04:52.886123 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:04:52 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:04:52.889470 18725 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:04:52.891079 18725 webserver.cc:460] Webserver started at http://127.2.205.131:45445/ using document root <none> and password file <none>
I20260929 03:04:52.891793 18725 fs_manager.cc:359] Metadata directory not provided
I20260929 03:04:52.891919 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:04:52.892222 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:04:52.895300 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "39229a97e8fc43f2bf387d5676800ea2"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "46c855e7b1104fb07a064f67a1c5838c"
server_key_iv: "c3a763acddb05dd1e4926ec78c14d469"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.896126 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "39229a97e8fc43f2bf387d5676800ea2"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "46c855e7b1104fb07a064f67a1c5838c"
server_key_iv: "c3a763acddb05dd1e4926ec78c14d469"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.901615 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 03:04:52.905444 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.907140 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:04:52.907307 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "39229a97e8fc43f2bf387d5676800ea2"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "46c855e7b1104fb07a064f67a1c5838c"
server_key_iv: "c3a763acddb05dd1e4926ec78c14d469"
server_key_version: "encryptionkey@0"
I20260929 03:04:52.907511 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:04:52.926837 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:04:52.931049 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:04:52.931317 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:04:52.932271 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:04:52.933645 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:04:52.933717 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.933826 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:04:52.933872 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:04:52.961704 18725 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:33529
I20260929 03:04:52.961752 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:33529 every 8 connection(s)
I20260929 03:04:52.963068 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:04:52.972718 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18725
I20260929 03:04:52.972914 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:04:52.973444 2870 external_mini_cluster.cc:1428] Setting key 6ce27fcd9b3a659a502c654d8befa9a6
Sep 29 03:04:52 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:04:52.983266 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:52.965512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:57729 (local address 127.2.205.190:41047)
0929 03:04:52.965706 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:52.965712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:52.967016 (+ 1304us) server_negotiation.cc:407] Connection header received
0929 03:04:52.968818 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:52.968824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:52.968926 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:52.969038 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:52.970508 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.971436 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.972973 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.973593 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.973724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:52.977568 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:52.977573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:52.977577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:52.977613 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:52.980158 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:52.980917 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:52.980922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:52.980925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:52.981085 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:52.981510 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:52.981514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:52.981523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:52.981818 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:52.981950 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:52.982686 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 03:04:52.982928 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:04:52.984249 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:52.966032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:57729)
0929 03:04:52.966833 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:52.966884 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:52.968529 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:52.969241 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:52.969248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:52.969737 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:04:52.970347 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:52.970365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.971560 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.971564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:52.972838 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:52.972850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.973829 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.973835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:52.974034 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:52.974706 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:04:52.974740 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:04:52.977338 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:04:52.980292 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:52.980301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:52.980319 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:52.980810 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:52.981189 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:04:52.981195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:04:52.981199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:04:52.981397 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:04:52.981940 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:04:52.981948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:04:52.982313 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 03:04:52.982552 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 03:04:52.983130 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":337,"threads_started":1}
I20260929 03:04:52.985796 18853 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:04:52.986106 18853 heartbeater.cc:464] Registering TS with master...
I20260929 03:04:52.986852 18853 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:04:52.988402 18397 ts_manager.cc:194] Registered new tserver with Master: 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
I20260929 03:04:52.988999 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:04:52.989306 18397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:57729
I20260929 03:04:52.999588 18586 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:04:53.006789 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:52.993373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.2.205.190:41047)
0929 03:04:52.993544 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:52.993551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:52.994028 (+ 477us) server_negotiation.cc:407] Connection header received
0929 03:04:52.994210 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:52.994217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:52.994290 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:52.994373 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:52.995834 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.996604 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.998013 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:52.998453 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:52.998572 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.001253 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:53.001259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:53.001263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:53.001302 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:53.004023 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.004806 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.004813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.004817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.005003 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.005412 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.005415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.005417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.005754 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:53.005960 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.006317 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.006505 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:04:53.011124 18397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:04:53.018285 18397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:04:53.051702 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.031209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.2.205.130:35907)
0929 03:04:53.044344 (+ 13135us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.044353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.044383 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:04:53.044487 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.044495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.044787 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.045585 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:53.045811 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.047034 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.049395 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.050554 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.050685 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.050893 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.051189 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.051353 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12984,"thread_start_us":12850,"threads_started":1}
I20260929 03:04:53.051764 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.031130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:35907 (local address 127.0.0.1:32894)
0929 03:04:53.032512 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:53.032565 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:53.032781 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:53.045010 (+ 12229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:53.045015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:53.045042 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:53.045415 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.045425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.047196 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.047202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.049248 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.049266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.050750 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.050757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.050940 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.050994 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:04:53.051137 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 03:04:53.051322 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":4,"thread_start_us":5830,"threads_started":1}
I20260929 03:04:53.052446 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.037642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.0.0.1:36530)
0929 03:04:53.044957 (+ 7315us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:53.044984 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:53.045115 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:53.045716 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:53.045720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:53.045741 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:53.046083 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.046094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.047822 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.047827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.049863 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.049872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.051527 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.051534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.051690 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.051727 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:04:53.051830 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:04:53.052054 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7161,"mutex_wait_us":1,"spinlock_wait_cycles":4864,"thread_start_us":6858,"threads_started":1}
I20260929 03:04:53.052740 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.037745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.2.205.131:33529)
0929 03:04:53.043207 (+ 5462us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.043217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.045073 (+ 1856us) server_negotiation.cc:407] Connection header received
0929 03:04:53.045280 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.045288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.045521 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.045709 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:53.046284 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.047677 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.050023 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.051079 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.051182 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.051392 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.052267 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.052430 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5279,"thread_start_us":5786,"threads_started":1}
I20260929 03:04:53.053912 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.032207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.2.205.129:34723)
0929 03:04:53.042755 (+ 10548us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.042767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.042806 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:04:53.042962 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.042972 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.043295 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.043557 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:53.044976 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.046653 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.050011 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.051307 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.051436 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.051657 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.052784 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.053609 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10311,"thread_start_us":11542,"threads_started":1}
I20260929 03:04:53.054581 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.031588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:57080)
0929 03:04:53.037238 (+ 5650us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:53.037258 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:53.037378 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:53.043880 (+ 6502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:53.043889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:53.043928 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:53.044531 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.044547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.046796 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.046801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.049834 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.049852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.052119 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.052129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.052321 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.052352 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:04:53.052439 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:04:53.054222 (+ 1783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5511,"mutex_wait_us":712,"thread_start_us":817,"threads_started":1}
I20260929 03:04:53.055749 18655 tablet_service.cc:1467] Processing CreateTablet for tablet 27fe97740a6242018556eaf792c350e2 (DEFAULT_TABLE table=test-table [id=599f556ed8ab4acab1609dc56efacdc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:53.055960 18788 tablet_service.cc:1467] Processing CreateTablet for tablet 27fe97740a6242018556eaf792c350e2 (DEFAULT_TABLE table=test-table [id=599f556ed8ab4acab1609dc56efacdc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:53.056954 18655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fe97740a6242018556eaf792c350e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:53.057406 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fe97740a6242018556eaf792c350e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:53.057912 18521 tablet_service.cc:1467] Processing CreateTablet for tablet 27fe97740a6242018556eaf792c350e2 (DEFAULT_TABLE table=test-table [id=599f556ed8ab4acab1609dc56efacdc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:04:53.059222 18521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fe97740a6242018556eaf792c350e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:04:53.067857 18871 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Bootstrap starting.
I20260929 03:04:53.071074 18872 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Bootstrap starting.
I20260929 03:04:53.071895 18871 tablet_bootstrap.cc:654] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:53.072554 18873 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Bootstrap starting.
I20260929 03:04:53.073688 18871 log.cc:824] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:53.074481 18872 tablet_bootstrap.cc:654] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:53.075361 18873 tablet_bootstrap.cc:654] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Neither blocks nor log segments found. Creating new log.
I20260929 03:04:53.075563 18872 log.cc:824] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:53.076431 18873 log.cc:824] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Log is configured to *not* fsync() on all Append() calls
I20260929 03:04:53.076997 18871 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: No bootstrap required, opened a new log
I20260929 03:04:53.077298 18871 ts_tablet_manager.cc:1397] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 03:04:53.079054 18873 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: No bootstrap required, opened a new log
I20260929 03:04:53.079298 18873 ts_tablet_manager.cc:1397] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 03:04:53.087103 18873 raft_consensus.cc:348] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.087493 18873 raft_consensus.cc:374] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:53.087590 18873 raft_consensus.cc:729] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309f1c09ef734c1080e50511298ac994, State: Initialized, Role: FOLLOWER
I20260929 03:04:53.087941 18872 tablet_bootstrap.cc:492] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: No bootstrap required, opened a new log
I20260929 03:04:53.087888 18871 raft_consensus.cc:348] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.088258 18872 ts_tablet_manager.cc:1397] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260929 03:04:53.088352 18871 raft_consensus.cc:374] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:53.088222 18873 consensus_queue.cc:260] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.088457 18871 raft_consensus.cc:729] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01319fab600141168ae2d62dbdf7b8cb, State: Initialized, Role: FOLLOWER
I20260929 03:04:53.089237 18871 consensus_queue.cc:260] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.090018 18873 ts_tablet_manager.cc:1428] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 03:04:53.090523 18720 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:04:53.091081 18871 ts_tablet_manager.cc:1428] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 03:04:53.099889 18872 raft_consensus.cc:348] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.100328 18872 raft_consensus.cc:374] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:04:53.100430 18872 raft_consensus.cc:729] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39229a97e8fc43f2bf387d5676800ea2, State: Initialized, Role: FOLLOWER
I20260929 03:04:53.101069 18872 consensus_queue.cc:260] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.102102 18853 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:04:53.102595 18872 ts_tablet_manager.cc:1428] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 03:04:53.120783 18878 raft_consensus.cc:482] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:04:53.121148 18878 raft_consensus.cc:504] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.122921 18878 leader_election.cc:288] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529), 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723)
I20260929 03:04:53.132242 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.123538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:58635 (local address 127.2.205.131:33529)
0929 03:04:53.123715 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.123720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.124415 (+ 695us) server_negotiation.cc:407] Connection header received
0929 03:04:53.124621 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.124626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.124707 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.124832 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:53.125691 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.126852 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.129994 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.131186 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.131362 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.131437 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.131845 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.131983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:04:53.132246 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.123430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:55729 (local address 127.2.205.129:34723)
0929 03:04:53.123665 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.123671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.124415 (+ 744us) server_negotiation.cc:407] Connection header received
0929 03:04:53.124620 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.124626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.124707 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.124835 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:04:53.125691 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.126852 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.129944 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.131190 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.131359 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.131437 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.131845 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.131983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 03:04:53.132714 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.123327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.2.205.130:55729)
0929 03:04:53.124277 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:53.124291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:53.124480 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:53.124853 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:53.124857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:53.124893 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:53.125469 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.125484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.127072 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.127079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.129771 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.129781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.131454 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.131460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.131657 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.131714 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 03:04:53.131812 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:04:53.131986 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":212,"spinlock_wait_cycles":39680,"thread_start_us":523,"threads_started":1}
I20260929 03:04:53.132784 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.123260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.2.205.130:58635)
0929 03:04:53.124268 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0929 03:04:53.124290 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:04:53.124480 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:04:53.124853 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:04:53.124857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:04:53.124890 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:04:53.125525 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.125534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.127075 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.127079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.129764 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:04:53.129782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.131453 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.131460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:04:53.131662 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.131723 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 03:04:53.131812 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:04:53.131986 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":225,"spinlock_wait_cycles":12672,"thread_start_us":606,"threads_started":1}
I20260929 03:04:53.133786 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fe97740a6242018556eaf792c350e2" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39229a97e8fc43f2bf387d5676800ea2" is_pre_election: true
I20260929 03:04:53.133845 18541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fe97740a6242018556eaf792c350e2" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f1c09ef734c1080e50511298ac994" is_pre_election: true
I20260929 03:04:53.134310 18808 raft_consensus.cc:2454] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 0.
I20260929 03:04:53.134342 18541 raft_consensus.cc:2454] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 0.
I20260929 03:04:53.135001 18610 leader_election.cc:302] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01319fab600141168ae2d62dbdf7b8cb, 309f1c09ef734c1080e50511298ac994; no voters:
I20260929 03:04:53.135422 18878 raft_consensus.cc:2781] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:04:53.135591 18878 raft_consensus.cc:482] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:04:53.135691 18878 raft_consensus.cc:3037] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:53.137975 18878 raft_consensus.cc:504] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.138653 18878 leader_election.cc:288] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 election: Requested vote from peers 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529), 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723)
I20260929 03:04:53.139253 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fe97740a6242018556eaf792c350e2" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39229a97e8fc43f2bf387d5676800ea2"
I20260929 03:04:53.139259 18541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fe97740a6242018556eaf792c350e2" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f1c09ef734c1080e50511298ac994"
I20260929 03:04:53.139456 18808 raft_consensus.cc:3037] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:53.139467 18541 raft_consensus.cc:3037] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:04:53.141809 18808 raft_consensus.cc:2454] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 1.
I20260929 03:04:53.141826 18541 raft_consensus.cc:2454] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 1.
I20260929 03:04:53.142299 18610 leader_election.cc:302] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01319fab600141168ae2d62dbdf7b8cb, 309f1c09ef734c1080e50511298ac994; no voters:
I20260929 03:04:53.142652 18878 raft_consensus.cc:2781] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:04:53.143126 18878 raft_consensus.cc:686] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 1 LEADER]: Becoming Leader. State: Replica: 01319fab600141168ae2d62dbdf7b8cb, State: Running, Role: LEADER
I20260929 03:04:53.143625 18878 consensus_queue.cc:237] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } }
I20260929 03:04:53.148154 18396 catalog_manager.cc:5495] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130). New cstate: current_term: 1 leader_uuid: "01319fab600141168ae2d62dbdf7b8cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } health_report { overall_health: UNKNOWN } } }
I20260929 03:04:53.165270 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.159950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.2.205.130:35907)
0929 03:04:53.160159 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.160166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.160185 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:53.160342 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.160346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.160442 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.160565 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:04:53.160991 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.161703 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.163130 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.163583 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.163690 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.164178 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:04:53.164669 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:04:53.164832 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.164931 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.165011 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:04:53.173213 18541 raft_consensus.cc:1261] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Refusing update from remote peer 01319fab600141168ae2d62dbdf7b8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:53.173213 18808 raft_consensus.cc:1261] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Refusing update from remote peer 01319fab600141168ae2d62dbdf7b8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:04:53.174297 18878 consensus_queue.cc:1035] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:53.174588 18882 consensus_queue.cc:1035] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:04:53.200075 18395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:53.200335 18395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:04:53.203890 18395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Sending DeleteTablet for 3 replicas of tablet 27fe97740a6242018556eaf792c350e2
I20260929 03:04:53.204907 18655 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fe97740a6242018556eaf792c350e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:53 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:32894
I20260929 03:04:53.205024 18788 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fe97740a6242018556eaf792c350e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:53 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36530
I20260929 03:04:53.205235 18521 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fe97740a6242018556eaf792c350e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:04:53 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:57080
I20260929 03:04:53.205380 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18458
I20260929 03:04:53.217486 18896 tablet_replica.cc:330] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: stopping tablet replica
I20260929 03:04:53.218034 18896 raft_consensus.cc:2229] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:53.218523 18894 tablet_replica.cc:330] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: stopping tablet replica
I20260929 03:04:53.218648 18896 raft_consensus.cc:2258] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:53.218988 18894 raft_consensus.cc:2229] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:04:53.219820 18894 raft_consensus.cc:2258] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:53.222711 18896 ts_tablet_manager.cc:1905] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:53.223323 18895 tablet_replica.cc:330] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: stopping tablet replica
I20260929 03:04:53.223793 18895 raft_consensus.cc:2229] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:04:53.226699 18894 ts_tablet_manager.cc:1905] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:53.227989 18895 raft_consensus.cc:2258] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:04:53.231091 18896 ts_tablet_manager.cc:1918] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:53.231251 18896 log.cc:1196] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/27fe97740a6242018556eaf792c350e2
I20260929 03:04:53.231750 18896 ts_tablet_manager.cc:1939] T 27fe97740a6242018556eaf792c350e2 P 309f1c09ef734c1080e50511298ac994: Deleting consensus metadata
I20260929 03:04:53.233723 18383 catalog_manager.cc:4841] TS 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723): tablet 27fe97740a6242018556eaf792c350e2 (table test-table [id=599f556ed8ab4acab1609dc56efacdc4]) successfully deleted
I20260929 03:04:53.235006 18894 ts_tablet_manager.cc:1918] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:53.235181 18894 log.cc:1196] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/27fe97740a6242018556eaf792c350e2
Sep 29 03:04:53 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:04:53.235724 18894 ts_tablet_manager.cc:1939] T 27fe97740a6242018556eaf792c350e2 P 01319fab600141168ae2d62dbdf7b8cb: Deleting consensus metadata
I20260929 03:04:53.237695 18381 catalog_manager.cc:4841] TS 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): tablet 27fe97740a6242018556eaf792c350e2 (table test-table [id=599f556ed8ab4acab1609dc56efacdc4]) successfully deleted
I20260929 03:04:53.242575 18895 ts_tablet_manager.cc:1905] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:04:53.244113 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.206556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.2.205.129:34723)
0929 03:04:53.206743 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.206750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.217492 (+ 10742us) server_negotiation.cc:407] Connection header received
0929 03:04:53.218020 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.218026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.218125 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.218252 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:53.222306 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.223366 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.224942 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.225483 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.225565 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.237544 (+ 11979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:53.237557 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:53.237571 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:53.237607 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:53.240392 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.241311 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.241322 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.241325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.241500 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.242026 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.242032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.242036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.242593 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:53.242965 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.243443 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.243724 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:04:53.244887 18581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:53.250386 18895 ts_tablet_manager.cc:1918] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:04:53.250586 18895 log.cc:1196] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/27fe97740a6242018556eaf792c350e2
I20260929 03:04:53.251003 18895 ts_tablet_manager.cc:1939] T 27fe97740a6242018556eaf792c350e2 P 39229a97e8fc43f2bf387d5676800ea2: Deleting consensus metadata
I20260929 03:04:53.252624 18381 catalog_manager.cc:4841] TS 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): tablet 27fe97740a6242018556eaf792c350e2 (table test-table [id=599f556ed8ab4acab1609dc56efacdc4]) successfully deleted
I20260929 03:04:53.380088 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18458
I20260929 03:04:53.395404 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18592
Sep 29 03:04:53 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:04:53.413895 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.396248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.2.205.130:35907)
0929 03:04:53.396658 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.396664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.396683 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:53.396777 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.396783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.396865 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.396979 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:53.398314 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.399479 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.401231 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.401895 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.402026 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.406233 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:53.406254 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:53.406277 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:53.406328 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:53.410411 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.411259 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.411271 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.411275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.411461 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.411959 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.411965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.411968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.412396 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:53.412810 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.413269 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.413498 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 03:04:53.414633 18715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:53.530411 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18592
I20260929 03:04:53.545729 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18725
Sep 29 03:04:53 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:04:53.560401 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:04:53.546456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.2.205.131:33529)
0929 03:04:53.546707 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 03:04:53.546713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:04:53.546732 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:04:53.546860 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:04:53.546864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:04:53.546930 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:04:53.547013 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:04:53.548335 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.549109 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.550463 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:04:53.550851 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:04:53.550961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:04:53.555239 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:04:53.555254 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:04:53.555269 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:04:53.555306 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:04:53.557757 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.558445 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.558459 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.558461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.558595 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:04:53.558945 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:04:53.558949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:04:53.558953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:04:53.559296 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:04:53.559570 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 03:04:53.559873 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 03:04:53.560083 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 03:04:53.560981 18848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:53.648906 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18725
I20260929 03:04:53.661742 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18365
I20260929 03:04:53.662793 18427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:04:53.773231 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18365
I20260929 03:05:13.791531 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--webserver_interface=127.2.205.190
--webserver_port=35375
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:14.102770 18902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:14.103103 18902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:14.103238 18902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:14.110597 18902 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:05:14.110698 18902 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:05:14.110742 18902 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:05:14.110812 18902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:05:14.110870 18902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:14.110955 18902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:14.111011 18902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:05:14.111089 18902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:05:14.121235 18902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:14.122562 18902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:14.123958 18902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:14.145562 18908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:14.145601 18911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:14.145622 18909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:14.145972 18902 server_base.cc:1034] running on GCE node
I20260929 03:05:14.147080 18902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:14.148648 18902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:14.149868 18902 hybrid_clock.cc:648] HybridClock initialized: now 1790651114149794 us; error 51 us; skew 500 ppm
I20260929 03:05:14.150238 18902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:14 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651114, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:14.157073 18902 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:05:14.158744 18902 webserver.cc:460] Webserver started at http://127.2.205.190:35375/ using document root <none> and password file <none>
I20260929 03:05:14.159503 18902 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:14.159621 18902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:14.165814 18902 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:05:14.168669 18918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:14.170051 18902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:05:14.170215 18902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "dd4ee9e42b734b2db0434270cbf6944d"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "03a897fb63009286ac8fb53797be5fcd"
server_key_iv: "839dda0b333917934fba4ea8dbe06c86"
server_key_version: "encryptionkey@0"
I20260929 03:05:14.171222 18902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:14.186940 18902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:14.191103 18902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:14.191361 18902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:14.208293 18902 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:41047
I20260929 03:05:14.208315 18969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:41047 every 8 connection(s)
I20260929 03:05:14.209764 18902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:05:14.212138 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18902
I20260929 03:05:14.212934 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:34723
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=35425
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:05:14.216774 18970 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:05:14.220223 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap starting.
I20260929 03:05:14.242484 18970 log.cc:824] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:14.254318 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:05:14.254807 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap complete.
I20260929 03:05:14.264159 18970 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:14.265053 18970 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Initialized, Role: FOLLOWER
I20260929 03:05:14.265687 18970 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:14.265915 18970 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:05:14.266026 18970 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:05:14.266185 18970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:05:14.269906 18970 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:14.270391 18970 leader_election.cc:302] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd4ee9e42b734b2db0434270cbf6944d; no voters:
I20260929 03:05:14.270988 18970 leader_election.cc:288] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:05:14.271135 18974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:05:14.272137 18974 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 LEADER]: Becoming Leader. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Running, Role: LEADER
I20260929 03:05:14.272668 18974 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:14.273160 18970 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:05:14.274983 18976 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd4ee9e42b734b2db0434270cbf6944d. Latest consensus state: current_term: 2 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:05:14.275329 18976 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:14.275127 18975 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:05:14.275482 18975 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:14.281661 18981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:05:14.290030 18981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:05:14.291684 18981 catalog_manager.cc:1259] Loaded cluster ID: d0f6cd848f974963ba9ea9a08c1ad299
I20260929 03:05:14.291786 18981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:05:14.296072 18981 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:05:14.297637 18981 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Loaded TSK: 0
I20260929 03:05:14.331665 18981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Generated new TSK 1
I20260929 03:05:14.332033 18981 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:05:14.568007 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:14.568306 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:14.568439 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:14.575901 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:14.576026 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:14.576184 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:05:14.586164 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=35425
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:14.587486 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:14.588845 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:14.603039 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:14.603068 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:14.603190 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:14.603749 18972 server_base.cc:1034] running on GCE node
I20260929 03:05:14.604332 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:14.605247 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:14.606432 18972 hybrid_clock.cc:648] HybridClock initialized: now 1790651114606367 us; error 61 us; skew 500 ppm
I20260929 03:05:14.606714 18972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:14 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651114, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:14.609850 18972 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:05:14.611256 18972 webserver.cc:460] Webserver started at http://127.2.205.129:35425/ using document root <none> and password file <none>
I20260929 03:05:14.611887 18972 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:14.612007 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:14.617268 18972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:05:14.619997 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:14.621447 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:05:14.621594 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "309f1c09ef734c1080e50511298ac994"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "ae040536fb312e9dbf841d89fd7a8f4a"
server_key_iv: "b18d8fe84f2f1084f3f1bacf4ec413ab"
server_key_version: "encryptionkey@0"
I20260929 03:05:14.622534 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:14.658711 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:14.662811 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:14.663064 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:14.663985 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:14.665360 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:14.665457 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:14.665563 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:14.665638 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:14.694782 18972 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34723
I20260929 03:05:14.694828 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34723 every 8 connection(s)
I20260929 03:05:14.696174 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:05:14.698830 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 18972
I20260929 03:05:14.699517 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:35907
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=39485
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:05:14 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651114, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:14.721200 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:14.698991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:58621 (local address 127.2.205.190:41047)
0929 03:05:14.699607 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:14.699623 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:14.700403 (+ 780us) server_negotiation.cc:407] Connection header received
0929 03:05:14.703076 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:14.703089 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:14.703446 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:14.703775 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:14.705653 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:14.706917 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:14.709301 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:14.710020 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:14.710160 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:14.714132 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:14.714142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:14.714156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:14.714207 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:14.716930 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:14.717790 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:14.717797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:14.717802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:14.717961 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:14.718406 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:14.718408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:14.718410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:14.718871 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:14.719011 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:14.719926 (+ 915us) server_negotiation.cc:299] Negotiation successful
0929 03:05:14.720244 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260929 03:05:14.721642 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:14.699112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:58621)
0929 03:05:14.700199 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:14.700259 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:14.702513 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:14.703986 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:14.703997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:14.704700 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:14.705463 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:14.705489 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:14.707084 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:14.707090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:14.709113 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:14.709128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:14.710250 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:14.710257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:14.710529 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:14.711198 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:14.711236 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:14.713828 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:14.717063 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:14.717072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:14.717087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:14.717663 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:14.718064 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:14.718069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:14.718072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:14.718311 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:14.718997 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:14.719005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:14.719430 (+ 425us) client_negotiation.cc:769] Sending connection context
0929 03:05:14.719762 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 03:05:14.720317 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":547,"threads_started":1}
I20260929 03:05:14.724303 19120 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:05:14.724630 19120 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:14.725489 19120 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:05:14.727710 18935 ts_manager.cc:194] Registered new tserver with Master: 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723)
I20260929 03:05:14.729238 18935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:58621
W20260929 03:05:15.011183 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:15.011456 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:15.011583 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:15.018591 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:15.018703 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:15.018855 19124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:05:15.028100 19124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:35907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=39485
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:15.029318 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:15.030617 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:15.042742 19130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:15.042938 19129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:15.043164 19132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:15.043452 19124 server_base.cc:1034] running on GCE node
I20260929 03:05:15.044029 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:15.044955 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:15.046177 19124 hybrid_clock.cc:648] HybridClock initialized: now 1790651115046130 us; error 39 us; skew 500 ppm
I20260929 03:05:15.046487 19124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:15 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:15.049434 19124 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:05:15.050771 19124 webserver.cc:460] Webserver started at http://127.2.205.130:39485/ using document root <none> and password file <none>
I20260929 03:05:15.051379 19124 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:15.051501 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:15.056535 19124 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:05:15.059019 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.060343 19124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:05:15.060496 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "01319fab600141168ae2d62dbdf7b8cb"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "0a7d676deddda61e81fc46cee276d93e"
server_key_iv: "c6c466f5f2b80ad68e5c38fa1fc5abf0"
server_key_version: "encryptionkey@0"
I20260929 03:05:15.061455 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:15.098838 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:15.102605 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:15.102842 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:15.103797 19124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:15.105060 19124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:15.105166 19124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.105269 19124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:15.105336 19124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.134570 19124 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:35907
I20260929 03:05:15.134622 19251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:35907 every 8 connection(s)
I20260929 03:05:15.135944 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:05:15.144050 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19124
I20260929 03:05:15.144685 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:33529
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=45445
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:05:15 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:15.155959 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.138370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:44393 (local address 127.2.205.190:41047)
0929 03:05:15.138571 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.138576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.139780 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 03:05:15.141651 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.141656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.141723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.141812 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.143347 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.144267 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.145929 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.146576 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.146804 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.150265 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.150272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.150276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.150314 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.152939 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.153756 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.153760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.153762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.153886 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.154329 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.154334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.154337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.154613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.154717 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.155533 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.155711 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:05:15.157232 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.138811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:44393)
0929 03:05:15.139608 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.139646 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.141391 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.142067 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.142088 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.142574 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:15.143190 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.143209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.144433 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.144437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.145787 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.145800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.146844 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.146849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.147073 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.147729 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:15.147751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:15.150030 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:15.153065 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:15.153074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:15.153090 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:15.153627 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:15.153986 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:15.153990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:15.153994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:15.154222 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:15.154719 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:15.154727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:15.155142 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.155422 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.155910 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":309,"threads_started":1}
I20260929 03:05:15.158648 19252 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:05:15.158936 19252 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:15.159660 19252 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:05:15.161096 18935 ts_manager.cc:194] Registered new tserver with Master: 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907)
I20260929 03:05:15.161899 18935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:44393
W20260929 03:05:15.458505 19255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:15.458789 19255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:15.458915 19255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:15.466143 19255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:15.466269 19255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:15.466416 19255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:05:15.476151 19255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=45445
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:15.477478 19255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:15.478780 19255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:15.491155 19260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:15.491592 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:15.491598 19263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:15.492563 19255 server_base.cc:1034] running on GCE node
I20260929 03:05:15.493172 19255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:15.494047 19255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:15.495247 19255 hybrid_clock.cc:648] HybridClock initialized: now 1790651115495196 us; error 42 us; skew 500 ppm
I20260929 03:05:15.495537 19255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:15 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:15.498479 19255 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:05:15.499756 19255 webserver.cc:460] Webserver started at http://127.2.205.131:45445/ using document root <none> and password file <none>
I20260929 03:05:15.500372 19255 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:15.500491 19255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:15.505517 19255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:05:15.507936 19270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.509176 19255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:05:15.509311 19255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "39229a97e8fc43f2bf387d5676800ea2"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "46c855e7b1104fb07a064f67a1c5838c"
server_key_iv: "c3a763acddb05dd1e4926ec78c14d469"
server_key_version: "encryptionkey@0"
I20260929 03:05:15.510177 19255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:15.534657 19255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:15.538582 19255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:15.538810 19255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:15.539749 19255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:15.541079 19255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:15.541180 19255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.541280 19255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:15.541347 19255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:15.570233 19255 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:33529
I20260929 03:05:15.570287 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:33529 every 8 connection(s)
I20260929 03:05:15.571622 19255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:05:15.578703 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19255
Sep 29 03:05:15 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:15.592689 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.574209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:35477 (local address 127.2.205.190:41047)
0929 03:05:15.574408 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.574414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.575654 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 03:05:15.577516 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.577521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.577595 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.577701 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.579278 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.580354 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.582032 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.582529 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.582636 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.586265 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.586270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.586274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.586303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.589414 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.590140 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.590147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.590150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.590295 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.590697 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.590702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.590706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.591052 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.591228 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.592162 (+ 934us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.592320 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:05:15.593816 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.574748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:35477)
0929 03:05:15.575483 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.575536 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.577249 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.577899 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.577907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.578456 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:15.579075 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.579096 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.580520 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.580525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.581872 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.581884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.582748 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.582755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.582998 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.583696 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:15.583720 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:15.586049 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:15.589545 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:15.589552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:15.589579 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:15.589984 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:15.590396 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:15.590400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:15.590402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:15.590591 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:15.591241 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:15.591249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:15.591722 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.591998 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.592493 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":272,"threads_started":1}
I20260929 03:05:15.593962 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.579607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.2.205.190:41047)
0929 03:05:15.580382 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.580388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.580403 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:05:15.580453 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.580457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.580527 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.580635 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.582166 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.582975 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.584474 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.584921 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.585018 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.587843 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.587849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.587854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.587915 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.591126 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.591888 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.591891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.591893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.592010 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.592409 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.592414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.592421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.592804 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.593032 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.593529 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.593692 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":532,"threads_started":1}
I20260929 03:05:15.595192 19383 heartbeater.cc:347] Connected to a master server at 127.2.205.190:41047
I20260929 03:05:15.595467 19383 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:15.596128 19383 heartbeater.cc:511] Master 127.2.205.190:41047 requested a full tablet report, sending...
I20260929 03:05:15.597442 18934 ts_manager.cc:194] Registered new tserver with Master: 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
I20260929 03:05:15.598264 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:35477
I20260929 03:05:15.608479 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:05:15.615016 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.609444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.2.205.190:41047)
0929 03:05:15.609647 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.609654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.610024 (+ 370us) server_negotiation.cc:407] Connection header received
0929 03:05:15.610200 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.610204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.610331 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.610447 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:05:15.610877 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.611614 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.613095 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.613559 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.613670 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.614219 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:05:15.614448 (+ 229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:05:15.614778 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57980: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 03:05:15.615155 19387 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57980: token expired
W20260929 03:05:15.615695 18856 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.205.190:41047)
I20260929 03:05:15.627643 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.616229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.2.205.190:41047)
0929 03:05:15.616402 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.616407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.616422 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:05:15.616461 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.616465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.616521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.616606 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.618030 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.618752 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.619990 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.620368 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.620451 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.622804 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.622809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.622812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.622844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.625380 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.625996 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.625999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.626002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.626148 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.626485 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.626488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.626490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.626791 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.626979 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.627203 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.627367 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:05:15.630909 18934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:05:15.637897 18934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:05:15.659482 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.648730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37394)
0929 03:05:15.649678 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.649722 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.649897 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.651340 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.651349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.651391 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.651875 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.651887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.654057 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.654063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.656830 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.656847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.658463 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.658473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.658707 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.658775 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.658908 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.659102 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":66,"thread_start_us":682,"threads_started":1}
I20260929 03:05:15.659673 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.649685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.2.205.130:35907)
0929 03:05:15.650739 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.650748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.650779 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:05:15.650878 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.650886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.651134 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.651319 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.652368 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.653990 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.656866 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.658335 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.658461 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.658678 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.659205 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.659371 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":956,"threads_started":1}
I20260929 03:05:15.659802 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.648964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.2.205.129:34723)
0929 03:05:15.650458 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.650467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.650494 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:05:15.650590 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.650598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.650865 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.651059 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.652115 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.653874 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.657039 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.658217 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.658899 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.659091 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.659157 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.659314 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":1217,"threads_started":1}
I20260929 03:05:15.660306 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.648969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:35907 (local address 127.0.0.1:38502)
0929 03:05:15.650395 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.650411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.650528 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.651524 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.651529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.651578 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.652014 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.652026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.654172 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.654177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.656691 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.656704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.658843 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.658850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.659012 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.659043 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.659891 (+ 848us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.660026 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":680,"spinlock_wait_cycles":6016,"thread_start_us":1186,"threads_started":1}
I20260929 03:05:15.663518 19187 tablet_service.cc:1467] Processing CreateTablet for tablet 09447ef54167460fa5c1ec4b337cae2c (DEFAULT_TABLE table=test-table [id=d621494df3544298a4c1a23b1254aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:15.663614 19055 tablet_service.cc:1467] Processing CreateTablet for tablet 09447ef54167460fa5c1ec4b337cae2c (DEFAULT_TABLE table=test-table [id=d621494df3544298a4c1a23b1254aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:15.665014 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09447ef54167460fa5c1ec4b337cae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:15.665023 19055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09447ef54167460fa5c1ec4b337cae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:15.670426 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.650542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.0.0.1:59062)
0929 03:05:15.655121 (+ 4579us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.655137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.655238 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.658335 (+ 3097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.658340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.658366 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.658838 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.658849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.661119 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.661124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.663903 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.663918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.669641 (+ 5723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.669647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.669826 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.669859 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.669950 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.670100 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4440,"thread_start_us":4300,"threads_started":1}
I20260929 03:05:15.671193 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.650971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.2.205.131:33529)
0929 03:05:15.657337 (+ 6366us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.657345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.657377 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:05:15.657463 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.657470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.657689 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.657875 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.659163 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.660995 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.664144 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.669015 (+ 4871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.669152 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.669368 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.670692 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.670858 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6188,"thread_start_us":6037,"threads_started":1}
I20260929 03:05:15.675261 19318 tablet_service.cc:1467] Processing CreateTablet for tablet 09447ef54167460fa5c1ec4b337cae2c (DEFAULT_TABLE table=test-table [id=d621494df3544298a4c1a23b1254aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:15.676694 19318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09447ef54167460fa5c1ec4b337cae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:15.679908 19399 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Bootstrap starting.
I20260929 03:05:15.682595 19400 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Bootstrap starting.
I20260929 03:05:15.682929 19399 tablet_bootstrap.cc:654] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:15.684262 19399 log.cc:824] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:15.686393 19400 tablet_bootstrap.cc:654] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:15.687381 19399 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: No bootstrap required, opened a new log
I20260929 03:05:15.687649 19399 ts_tablet_manager.cc:1397] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260929 03:05:15.687743 19400 log.cc:824] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:15.690047 19400 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: No bootstrap required, opened a new log
I20260929 03:05:15.690287 19400 ts_tablet_manager.cc:1397] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 03:05:15.691751 19403 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Bootstrap starting.
I20260929 03:05:15.695557 19403 tablet_bootstrap.cc:654] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:15.696741 19403 log.cc:824] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:15.697577 19399 raft_consensus.cc:348] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.698091 19399 raft_consensus.cc:374] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:15.698242 19399 raft_consensus.cc:729] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309f1c09ef734c1080e50511298ac994, State: Initialized, Role: FOLLOWER
I20260929 03:05:15.698999 19399 consensus_queue.cc:260] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.700343 19400 raft_consensus.cc:348] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.700816 19400 raft_consensus.cc:374] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:15.700922 19400 raft_consensus.cc:729] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01319fab600141168ae2d62dbdf7b8cb, State: Initialized, Role: FOLLOWER
I20260929 03:05:15.701658 19400 consensus_queue.cc:260] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.708088 19406 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:15.708170 19252 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:05:15.710204 19400 ts_tablet_manager.cc:1428] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Time spent starting tablet: real 0.020s user 0.016s sys 0.001s
I20260929 03:05:15.710633 19406 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.713867 19120 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:05:15.714553 19399 ts_tablet_manager.cc:1428] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Time spent starting tablet: real 0.027s user 0.010s sys 0.016s
I20260929 03:05:15.715457 19403 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: No bootstrap required, opened a new log
I20260929 03:05:15.715749 19403 ts_tablet_manager.cc:1397] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.007s
I20260929 03:05:15.718731 19406 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
I20260929 03:05:15.720741 19405 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:15.721001 19405 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.722776 19405 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
I20260929 03:05:15.726277 19403 raft_consensus.cc:348] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.726732 19403 raft_consensus.cc:374] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:15.726897 19403 raft_consensus.cc:729] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39229a97e8fc43f2bf387d5676800ea2, State: Initialized, Role: FOLLOWER
I20260929 03:05:15.727676 19403 consensus_queue.cc:260] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.728572 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.719254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:34045 (local address 127.2.205.131:33529)
0929 03:05:15.719450 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.719459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.720605 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 03:05:15.720917 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.720924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.721015 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.721118 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.721873 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.723015 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.725802 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.727201 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.727342 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.727421 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.728140 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.728287 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:05:15.728699 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.719402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:32851 (local address 127.2.205.129:34723)
0929 03:05:15.719577 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.719582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.721468 (+ 1886us) server_negotiation.cc:407] Connection header received
0929 03:05:15.721715 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.721720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.721787 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.721842 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.722574 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.723649 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.726098 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.727315 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.727431 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.727497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.728267 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.728410 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:05:15.729496 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.718909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.2.205.130:34045)
0929 03:05:15.720501 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.720519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.720712 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.721311 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.721316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.721341 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.721687 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.721700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.723208 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.723212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.725567 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.725586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.727723 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.727730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.727934 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.727973 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.728879 (+ 906us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.729031 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"mutex_wait_us":1,"thread_start_us":4795,"threads_started":1}
I20260929 03:05:15.729785 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.719008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.2.205.130:32851)
0929 03:05:15.721373 (+ 2365us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.721393 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.721561 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.722021 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.722026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.722048 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.722401 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.722412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.723867 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.723872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.725929 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.725943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.727814 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.727820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.728078 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.728111 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.729100 (+ 989us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.729255 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2173,"thread_start_us":1229,"threads_started":1}
I20260929 03:05:15.730569 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39229a97e8fc43f2bf387d5676800ea2" is_pre_election: true
I20260929 03:05:15.730701 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f1c09ef734c1080e50511298ac994" is_pre_election: true
I20260929 03:05:15.731112 19338 raft_consensus.cc:2454] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 0.
I20260929 03:05:15.731096 19075 raft_consensus.cc:2454] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 0.
I20260929 03:05:15.730758 19383 heartbeater.cc:503] Master 127.2.205.190:41047 was elected leader, sending a full tablet report...
I20260929 03:05:15.731781 19403 ts_tablet_manager.cc:1428] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260929 03:05:15.731820 19142 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01319fab600141168ae2d62dbdf7b8cb, 309f1c09ef734c1080e50511298ac994; no voters:
I20260929 03:05:15.732231 19406 raft_consensus.cc:2781] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:05:15.732398 19406 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:05:15.732489 19406 raft_consensus.cc:3037] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:15.736192 19406 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.737289 19406 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 election: Requested vote from peers 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
I20260929 03:05:15.737394 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39229a97e8fc43f2bf387d5676800ea2"
I20260929 03:05:15.737564 19338 raft_consensus.cc:3037] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:15.737855 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "01319fab600141168ae2d62dbdf7b8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f1c09ef734c1080e50511298ac994"
I20260929 03:05:15.738066 19075 raft_consensus.cc:3037] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:15.740643 19338 raft_consensus.cc:2454] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 1.
I20260929 03:05:15.741057 19140 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2; no voters:
I20260929 03:05:15.741330 19075 raft_consensus.cc:2454] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01319fab600141168ae2d62dbdf7b8cb in term 1.
I20260929 03:05:15.741366 19406 raft_consensus.cc:2781] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:05:15.741523 19406 raft_consensus.cc:686] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 1 LEADER]: Becoming Leader. State: Replica: 01319fab600141168ae2d62dbdf7b8cb, State: Running, Role: LEADER
I20260929 03:05:15.742013 19406 consensus_queue.cc:237] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:15.742558 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.723516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:41917 (local address 127.2.205.130:35907)
0929 03:05:15.723701 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.723706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.734717 (+ 11011us) server_negotiation.cc:407] Connection header received
0929 03:05:15.735885 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.735890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.735958 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.736035 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.736561 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.737475 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.740221 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.741492 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.741623 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.741712 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.742256 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.742380 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:05:15.742667 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.723193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:35907 (local address 127.2.205.129:41917)
0929 03:05:15.734612 (+ 11419us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.734631 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.734759 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.736071 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.736085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.736115 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.736443 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.736455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.737779 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.737783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.740029 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.740043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.741888 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.741894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.742082 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.742115 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.742208 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.742328 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11283,"thread_start_us":11672,"threads_started":1}
I20260929 03:05:15.743448 19207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "309f1c09ef734c1080e50511298ac994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01319fab600141168ae2d62dbdf7b8cb" is_pre_election: true
I20260929 03:05:15.746466 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.735296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.2.205.129:40017)
0929 03:05:15.737840 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.737858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:15.738031 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:15.738494 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:15.738499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:15.738529 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:15.738973 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.738986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.740461 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.740466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.744371 (+ 3905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:15.744388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.745825 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.745831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:15.745986 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.746017 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:05:15.746120 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:05:15.746239 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2409,"thread_start_us":2260,"threads_started":1}
I20260929 03:05:15.746521 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.735359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:40017 (local address 127.2.205.131:33529)
0929 03:05:15.735520 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.735525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.737934 (+ 2409us) server_negotiation.cc:407] Connection header received
0929 03:05:15.738189 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.738194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.738258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.738320 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:15.739212 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.740309 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.744532 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.745607 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.745764 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.745844 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.746162 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.746285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:05:15.747102 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09447ef54167460fa5c1ec4b337cae2c" candidate_uuid: "309f1c09ef734c1080e50511298ac994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39229a97e8fc43f2bf387d5676800ea2" is_pre_election: true
I20260929 03:05:15.747334 19338 raft_consensus.cc:2379] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 309f1c09ef734c1080e50511298ac994 in current term 1: Already voted for candidate 01319fab600141168ae2d62dbdf7b8cb in this term.
I20260929 03:05:15.746866 18935 catalog_manager.cc:5495] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130). New cstate: current_term: 1 leader_uuid: "01319fab600141168ae2d62dbdf7b8cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20260929 03:05:15.747870 19008 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:15.748240 19405 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 03:05:15.765829 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.760831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.2.205.130:35907)
0929 03:05:15.760967 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.760971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.760988 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:05:15.761182 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.761186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.761260 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.761371 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:05:15.761668 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.762405 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.763801 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.764240 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.764345 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.764728 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:05:15.765315 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:05:15.765435 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.765499 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.765561 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:05:15.773351 19338 raft_consensus.cc:1261] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Refusing update from remote peer 01319fab600141168ae2d62dbdf7b8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:15.773397 19075 raft_consensus.cc:1261] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Refusing update from remote peer 01319fab600141168ae2d62dbdf7b8cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:15.774145 19407 consensus_queue.cc:1035] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:05:15.774506 19406 consensus_queue.cc:1035] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:05:15.784019 19414 mvcc.cc:205] Tried to move back new op lower bound from 7334506970198310912 to 7334506970084106240. Current Snapshot: MvccSnapshot[applied={T|T < 7334506970198310912}]
I20260929 03:05:15.787231 19416 mvcc.cc:205] Tried to move back new op lower bound from 7334506970198310912 to 7334506970084106240. Current Snapshot: MvccSnapshot[applied={T|T < 7334506970198310912 or (T in {7334506970198310912})}]
I20260929 03:05:15.800920 18934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:15.801164 18934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:15.804462 18934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Sending DeleteTablet for 3 replicas of tablet 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:15.805538 19055 tablet_service.cc:1514] Processing DeleteTablet for tablet 09447ef54167460fa5c1ec4b337cae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:15 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:37394
I20260929 03:05:15.805663 19187 tablet_service.cc:1514] Processing DeleteTablet for tablet 09447ef54167460fa5c1ec4b337cae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:15 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:38502
I20260929 03:05:15.805727 19318 tablet_service.cc:1514] Processing DeleteTablet for tablet 09447ef54167460fa5c1ec4b337cae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:15 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:59062
I20260929 03:05:15.806646 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18972
I20260929 03:05:15.818243 19424 tablet_replica.cc:330] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: stopping tablet replica
I20260929 03:05:15.818818 19424 raft_consensus.cc:2229] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:05:15.819430 19424 raft_consensus.cc:2258] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:15.821295 19425 tablet_replica.cc:330] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: stopping tablet replica
I20260929 03:05:15.821318 19426 tablet_replica.cc:330] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: stopping tablet replica
I20260929 03:05:15.821800 19425 raft_consensus.cc:2229] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:05:15.822144 19426 raft_consensus.cc:2229] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:05:15.822629 19426 raft_consensus.cc:2258] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:15.823449 19424 ts_tablet_manager.cc:1905] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:05:15.824136 19425 raft_consensus.cc:2258] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:15.824494 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.807468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.2.205.129:34723)
0929 03:05:15.807654 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.807661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.807731 (+ 70us) server_negotiation.cc:407] Connection header received
0929 03:05:15.807929 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.807935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.808020 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.808123 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.809970 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.811231 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.813253 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.813933 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.814050 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.817709 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.817729 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.817748 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.817794 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.820937 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.821912 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.821921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.821925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.822098 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.822543 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.822549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.822552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.823042 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.823325 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.823907 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.824128 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:05:15.824673 19426 ts_tablet_manager.cc:1905] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:05:15.825824 19115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:15.830835 19425 ts_tablet_manager.cc:1905] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:05:15.832232 19426 ts_tablet_manager.cc:1918] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:05:15.832376 19426 log.cc:1196] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:15.832894 19426 ts_tablet_manager.cc:1939] T 09447ef54167460fa5c1ec4b337cae2c P 39229a97e8fc43f2bf387d5676800ea2: Deleting consensus metadata
I20260929 03:05:15.834610 18919 catalog_manager.cc:4841] TS 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): tablet 09447ef54167460fa5c1ec4b337cae2c (table test-table [id=d621494df3544298a4c1a23b1254aa93]) successfully deleted
I20260929 03:05:15.836799 19425 ts_tablet_manager.cc:1918] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:05:15.836947 19425 log.cc:1196] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:15.837361 19425 ts_tablet_manager.cc:1939] T 09447ef54167460fa5c1ec4b337cae2c P 01319fab600141168ae2d62dbdf7b8cb: Deleting consensus metadata
I20260929 03:05:15.838626 18919 catalog_manager.cc:4841] TS 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): tablet 09447ef54167460fa5c1ec4b337cae2c (table test-table [id=d621494df3544298a4c1a23b1254aa93]) successfully deleted
I20260929 03:05:15.924875 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18972
W20260929 03:05:15.939771 18921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:34723 (error 108)
I20260929 03:05:15.939934 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 19124
I20260929 03:05:15.941994 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.941563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37424)
0929 03:05:15.941772 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.941858 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 03:05:15.942237 18921 catalog_manager.cc:4584] TS 309f1c09ef734c1080e50511298ac994 (127.2.205.129:34723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09447ef54167460fa5c1ec4b337cae2c: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
I20260929 03:05:15.955506 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.941110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.2.205.130:35907)
0929 03:05:15.941304 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:15.941310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:15.941735 (+ 425us) server_negotiation.cc:407] Connection header received
0929 03:05:15.941825 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:15.941831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:15.941913 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:15.942012 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:15.943509 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.944635 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.945972 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:15.946636 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:15.946751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:15.949228 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:15.949243 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:15.949262 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:15.949316 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:15.952883 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.953505 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.953514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.953517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.953695 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:15.954015 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:15.954019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:15.954023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:15.954463 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:15.954759 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:15.954911 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:05:15.955140 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:05:15.956312 19247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:15.995946 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.995385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37432)
0929 03:05:15.995623 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.995750 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 03:05:15.996982 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:15.996581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37438)
0929 03:05:15.996723 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:15.996842 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:05:16.057585 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:16.056998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37444)
0929 03:05:16.057266 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:16.057379 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 03:05:16.058622 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:16.058265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37452)
0929 03:05:16.058405 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:16.058492 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:05:16.067009 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 19124
I20260929 03:05:16.081985 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 19255
I20260929 03:05:16.098836 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:16.082773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.2.205.131:33529)
0929 03:05:16.082979 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:16.082986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:16.083007 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:05:16.083115 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:16.083121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:16.083193 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:16.083287 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:16.084698 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:16.085723 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:16.087452 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:16.088074 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:16.088214 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:16.092037 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:16.092057 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:16.092077 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:16.092143 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:16.095471 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:16.096502 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:16.096513 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:16.096516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:16.096674 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:16.097231 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:16.097236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:16.097240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:16.097686 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:16.097962 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:16.098288 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:05:16.098504 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:05:16.099512 19378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:16.139151 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:16.138685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37468)
0929 03:05:16.138889 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:16.138983 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 03:05:16.140187 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:16.139806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:34723 (local address 127.0.0.1:37484)
0929 03:05:16.139997 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:16.140070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 03:05:16.196271 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 19255
I20260929 03:05:16.211758 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 18902
I20260929 03:05:16.213131 18965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:16.327622 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 18902
I20260929 03:05:36.346452 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--webserver_interface=127.2.205.190
--webserver_port=35375
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:36.681988 19432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:36.682348 19432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:36.682489 19432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:36.690066 19432 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:05:36.690187 19432 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:05:36.690248 19432 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:05:36.690320 19432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:05:36.690394 19432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:36.690512 19432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:36.690578 19432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:05:36.690652 19432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:05:36.700949 19432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:36.702322 19432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:36.703704 19432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:36.715480 19440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:36.715523 19438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:36.715703 19437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:36.716693 19432 server_base.cc:1034] running on GCE node
I20260929 03:05:36.717592 19432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:36.719136 19432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:36.720381 19432 hybrid_clock.cc:648] HybridClock initialized: now 1790651136720315 us; error 40 us; skew 500 ppm
I20260929 03:05:36.720714 19432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:36 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:36.724048 19432 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:05:36.725618 19432 webserver.cc:460] Webserver started at http://127.2.205.190:35375/ using document root <none> and password file <none>
I20260929 03:05:36.726354 19432 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:36.726446 19432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:36.732108 19432 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 03:05:36.735139 19447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:36.736595 19432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:05:36.736802 19432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "dd4ee9e42b734b2db0434270cbf6944d"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "03a897fb63009286ac8fb53797be5fcd"
server_key_iv: "839dda0b333917934fba4ea8dbe06c86"
server_key_version: "encryptionkey@0"
I20260929 03:05:36.737820 19432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:36.756518 19432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:36.760713 19432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:36.760980 19432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:36.778102 19432 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:41047
I20260929 03:05:36.778132 19498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:41047 every 8 connection(s)
I20260929 03:05:36.779697 19432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:05:36.787045 19499 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:05:36.788928 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19432
I20260929 03:05:36.789755 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:34723
--local_ip_for_outbound_sockets=127.2.205.129
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=35425
--webserver_interface=127.2.205.129
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:05:36.790818 19499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap starting.
I20260929 03:05:36.812742 19499 log.cc:824] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:36.829137 19499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:05:36.829695 19499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Bootstrap complete.
I20260929 03:05:36.839116 19499 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:36.840353 19499 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Initialized, Role: FOLLOWER
I20260929 03:05:36.841250 19499 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:36.841514 19499 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:05:36.841650 19499 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:05:36.841825 19499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:05:36.846154 19499 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:36.846699 19499 leader_election.cc:302] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd4ee9e42b734b2db0434270cbf6944d; no voters:
I20260929 03:05:36.847352 19499 leader_election.cc:288] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:05:36.847484 19503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:05:36.848657 19503 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [term 3 LEADER]: Becoming Leader. State: Replica: dd4ee9e42b734b2db0434270cbf6944d, State: Running, Role: LEADER
I20260929 03:05:36.849345 19503 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } }
I20260929 03:05:36.849643 19499 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:05:36.851732 19505 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:05:36.852097 19505 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:36.852641 19504 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd4ee9e42b734b2db0434270cbf6944d. Latest consensus state: current_term: 3 leader_uuid: "dd4ee9e42b734b2db0434270cbf6944d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4ee9e42b734b2db0434270cbf6944d" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 41047 } } }
I20260929 03:05:36.852934 19504 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:36.862547 19510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:05:36.871063 19510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:05:36.872499 19510 catalog_manager.cc:1259] Loaded cluster ID: d0f6cd848f974963ba9ea9a08c1ad299
I20260929 03:05:36.872886 19510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:05:36.877031 19510 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:05:36.878939 19510 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Loaded TSK: 0
I20260929 03:05:36.879180 19510 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd4ee9e42b734b2db0434270cbf6944d: Loaded TSK: 1
F20260929 03:05:36.880409 19510 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ab0c059ad google::LogMessage::Fail() at ??:0
@ 0x7f3ab0c0888a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ab0c054de google::LogMessage::Flush() at ??:0
@ 0x7f3ab0c06a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ab49697d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3ab495873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3ac66341c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3ac6633783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3ac66bf2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3ac66bf0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3ac67095e9 std::function<>::operator()() at ??:0
@ 0x7f3ac6637cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3ac6635c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ac66bd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3ac66bd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260929 03:05:36.937356 19523 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f3ab3851345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ab385491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3ab3854799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3ab382feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3ab6b096db start_thread at ??:0
@ 0x7f3aafa9f61f clone at ??:0
W20260929 03:05:37.191044 19501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:37.191355 19501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:37.191502 19501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:37.199172 19501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:37.199352 19501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:37.199527 19501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:05:37.209831 19501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=35425
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:37.211241 19501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:37.212666 19501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:37.226948 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:37.226971 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:37.226876 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:37.227738 19501 server_base.cc:1034] running on GCE node
I20260929 03:05:37.228380 19501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:37.229436 19501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:37.230646 19501 hybrid_clock.cc:648] HybridClock initialized: now 1790651137230590 us; error 41 us; skew 500 ppm
I20260929 03:05:37.231424 19501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:37 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:37.236791 19501 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:05:37.240365 19501 webserver.cc:460] Webserver started at http://127.2.205.129:35425/ using document root <none> and password file <none>
I20260929 03:05:37.241333 19501 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:37.241488 19501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:37.248102 19501 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 03:05:37.251782 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:37.253413 19501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 03:05:37.253554 19501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "309f1c09ef734c1080e50511298ac994"
format_stamp: "Formatted at 2026-09-29 03:04:51 on dist-test-slave-530n"
server_key: "ae040536fb312e9dbf841d89fd7a8f4a"
server_key_iv: "b18d8fe84f2f1084f3f1bacf4ec413ab"
server_key_version: "encryptionkey@0"
I20260929 03:05:37.254492 19501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:37.282488 19501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:37.286864 19501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:37.287124 19501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:37.288148 19501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:37.290041 19563 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 03:05:37.295802 19501 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:05:37.295908 19501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 03:05:37.296025 19501 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:05:37.299125 19501 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:05:37.299263 19501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 03:05:37.299713 19563 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Bootstrap starting.
I20260929 03:05:37.307888 19563 log.cc:824] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:37.322029 19563 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:05:37.322633 19563 tablet_bootstrap.cc:492] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Bootstrap complete.
I20260929 03:05:37.323225 19563 ts_tablet_manager.cc:1397] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20260929 03:05:37.331296 19563 raft_consensus.cc:348] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:37.333057 19563 raft_consensus.cc:729] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309f1c09ef734c1080e50511298ac994, State: Initialized, Role: FOLLOWER
I20260929 03:05:37.333856 19563 consensus_queue.cc:260] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:37.335412 19563 ts_tablet_manager.cc:1428] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 03:05:37.340255 19501 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:34723
I20260929 03:05:37.340292 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:34723 every 8 connection(s)
I20260929 03:05:37.341496 19501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:05:37.346125 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.344127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:49183)
0929 03:05:37.344932 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.345208 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":288,"threads_started":1}
I20260929 03:05:37.346776 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19501
W20260929 03:05:37.346891 19672 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:41047: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
I20260929 03:05:37.347473 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:35907
--local_ip_for_outbound_sockets=127.2.205.130
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=39485
--webserver_interface=127.2.205.130
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:05:37.347854 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.347446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:43853)
0929 03:05:37.347648 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.347737 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 03:05:37.348827 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.348415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:53241)
0929 03:05:37.348569 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.348647 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:05:37.349079 19672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:05:37.677852 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:37.678126 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:37.678228 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:37.685530 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:37.685678 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:37.685803 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:05:37.696355 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:35907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=39485
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:37.697728 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:37.699087 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:37.712742 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:37.712841 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:37.712811 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:37.713563 19675 server_base.cc:1034] running on GCE node
I20260929 03:05:37.714215 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:37.715303 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:37.716547 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790651137716482 us; error 49 us; skew 500 ppm
I20260929 03:05:37.716950 19675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:37 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:37.720256 19675 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:05:37.721961 19675 webserver.cc:460] Webserver started at http://127.2.205.130:39485/ using document root <none> and password file <none>
I20260929 03:05:37.722669 19675 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:37.722802 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:37.728588 19675 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 03:05:37.731566 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:37.733057 19675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:05:37.733227 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "01319fab600141168ae2d62dbdf7b8cb"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "0a7d676deddda61e81fc46cee276d93e"
server_key_iv: "c6c466f5f2b80ad68e5c38fa1fc5abf0"
server_key_version: "encryptionkey@0"
I20260929 03:05:37.734246 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:37.751698 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:37.755628 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:37.755894 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:37.756917 19675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:37.758221 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:37.758313 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:37.758435 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:37.758513 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:37.788172 19675 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:35907
I20260929 03:05:37.788262 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:35907 every 8 connection(s)
I20260929 03:05:37.789747 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:05:37.791200 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19675
I20260929 03:05:37.791937 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:33529
--local_ip_for_outbound_sockets=127.2.205.131
--tserver_master_addrs=127.2.205.190:41047
--webserver_port=45445
--webserver_interface=127.2.205.131
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:05:37.795975 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.793669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:54119)
0929 03:05:37.794507 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.794777 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":309,"threads_started":1}
W20260929 03:05:37.796818 19803 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:41047: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
I20260929 03:05:37.797855 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.797458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:34321)
0929 03:05:37.797619 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.797719 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:05:37.798811 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:37.798451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:42975)
0929 03:05:37.798592 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:37.798679 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 03:05:37.799093 19803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:05:38.117548 19806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:38.117856 19806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:38.118002 19806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:38.125360 19806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:38.125497 19806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:38.125654 19806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:05:38.135730 19806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=45445
--tserver_master_addrs=127.2.205.190:41047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:38.137172 19806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:38.138590 19806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:38.152176 19811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:38.152202 19814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:38.152135 19812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:38.152961 19806 server_base.cc:1034] running on GCE node
I20260929 03:05:38.153549 19806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:38.154515 19806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:38.155699 19806 hybrid_clock.cc:648] HybridClock initialized: now 1790651138155643 us; error 48 us; skew 500 ppm
I20260929 03:05:38.155989 19806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:38 dist-test-slave-530n krb5kdc[18352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:38.159080 19806 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:05:38.160619 19806 webserver.cc:460] Webserver started at http://127.2.205.131:45445/ using document root <none> and password file <none>
I20260929 03:05:38.161417 19806 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:38.161535 19806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:38.167415 19806 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 03:05:38.170506 19821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:38.172003 19806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:05:38.172169 19806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "39229a97e8fc43f2bf387d5676800ea2"
format_stamp: "Formatted at 2026-09-29 03:04:52 on dist-test-slave-530n"
server_key: "46c855e7b1104fb07a064f67a1c5838c"
server_key_iv: "c3a763acddb05dd1e4926ec78c14d469"
server_key_version: "encryptionkey@0"
I20260929 03:05:38.173251 19806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:38.195789 19806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:38.200474 19806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:38.200788 19806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:38.201887 19806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:38.203462 19806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:38.203568 19806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:38.203691 19806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:38.203768 19806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:38.234652 19806 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:33529
I20260929 03:05:38.234721 19933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:33529 every 8 connection(s)
I20260929 03:05:38.236320 19806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:05:38.241585 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.239609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:58187)
0929 03:05:38.240368 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.240606 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":283,"threads_started":1}
W20260929 03:05:38.242303 19934 heartbeater.cc:644] Failed to heartbeat to 127.2.205.190:41047 (0 consecutive failures): Network error: Failed to ping master at 127.2.205.190:41047: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
I20260929 03:05:38.243152 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.242806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:41399)
0929 03:05:38.242959 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.243045 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:05:38.243988 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.243692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:52611)
0929 03:05:38.243811 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.243884 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:05:38.244199 19934 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:05:38.246302 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 19806
I20260929 03:05:38.350337 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.349629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:42071)
0929 03:05:38.350054 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.350138 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":193,"threads_started":1}
I20260929 03:05:38.800489 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.799661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:41199)
0929 03:05:38.800140 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.800261 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260929 03:05:38.930034 19940 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:38.930416 19940 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:38.932117 19940 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
Sep 29 03:05:38 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
Sep 29 03:05:38 dist-test-slave-530n krb5kdc[18352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:05:38.955889 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.933263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:45491 (local address 127.2.205.131:33529)
0929 03:05:38.934243 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:38.934259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:38.934820 (+ 561us) server_negotiation.cc:407] Connection header received
0929 03:05:38.936318 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:38.936327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:38.936787 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:38.937129 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:38.939363 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.940544 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.942725 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.943431 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.943576 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:38.948096 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:38.948108 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:38.948127 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:38.948184 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:38.951678 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:38.952656 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:38.952665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:38.952675 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:38.952922 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:38.953830 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:38.953834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:38.953837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:38.954431 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:38.954611 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:38.955134 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 03:05:38.955507 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":647,"threads_started":1}
I20260929 03:05:38.955911 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.932337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:35907 (local address 127.2.205.129:34099)
0929 03:05:38.932796 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.932835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:38.933430 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:38.937349 (+ 3919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:38.937362 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:38.938379 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:38.939139 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:38.939169 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.940733 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.940737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:38.942484 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:38.942498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.943622 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.943628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:38.943868 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:38.944635 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:38.944678 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:38.947551 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:38.951801 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:38.951811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:38.951830 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:38.952512 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:38.953032 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:38.953036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:38.953039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:38.953312 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:38.954234 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:38.954242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:38.954593 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 03:05:38.954882 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 03:05:38.955431 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":90,"spinlock_wait_cycles":43392,"thread_start_us":211,"threads_started":1}
I20260929 03:05:38.955943 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.932695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:34099 (local address 127.2.205.130:35907)
0929 03:05:38.934616 (+ 1921us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:38.934632 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:38.934693 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:05:38.936363 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:38.936372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:38.936783 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:38.937134 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:38.939355 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.940548 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.942772 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.943430 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.943611 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:38.947848 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:38.947856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:38.947876 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:38.947934 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:38.951637 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:38.952658 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:38.952665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:38.952675 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:38.952917 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:38.953448 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:38.953453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:38.953456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:38.954071 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:38.954230 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:38.955173 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 03:05:38.955521 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1734,"thread_start_us":1530,"threads_started":1}
I20260929 03:05:38.955983 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:38.932984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:33529 (local address 127.2.205.129:45491)
0929 03:05:38.934689 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:38.934707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:38.934840 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:38.937349 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:38.937362 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:38.938379 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:38.939150 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:38.939169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.940740 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.940765 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:38.942532 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:38.942548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:38.943924 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:38.943930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:38.944087 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:38.944675 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:38.944694 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:38.947798 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:38.951801 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:38.951811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:38.951830 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:38.952512 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:38.953193 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:38.953197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:38.953201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:38.953368 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:38.954588 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:38.954592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:38.954720 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 03:05:38.954882 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 03:05:38.955570 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588,"mutex_wait_us":359,"spinlock_wait_cycles":6528,"thread_start_us":1405,"threads_started":1}
W20260929 03:05:38.959043 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
W20260929 03:05:38.959342 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:38.959457 19557 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:38.959841 19940 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:05:39.245795 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:39.244850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:39659)
0929 03:05:39.245369 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:39.245538 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":249,"threads_started":1}
I20260929 03:05:39.351567 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:39.351163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:51859)
0929 03:05:39.351344 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:39.351432 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 03:05:39.802120 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:39.801402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:44519)
0929 03:05:39.801864 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:39.801951 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260929 03:05:40.247401 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:40.246729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:49467)
0929 03:05:40.247167 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:40.247250 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":236,"threads_started":1}
I20260929 03:05:40.353237 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:40.352434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:51263)
0929 03:05:40.352941 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:40.353037 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":228,"threads_started":1}
I20260929 03:05:40.494747 19949 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:40.494940 19949 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:40.495541 19949 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
W20260929 03:05:40.496590 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
W20260929 03:05:40.496860 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:40.496958 19557 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:40.497212 19949 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:05:40.803680 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:40.802964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:43539)
0929 03:05:40.803424 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:40.803508 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260929 03:05:41.248943 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:41.248227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:53339)
0929 03:05:41.248644 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:41.248766 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260929 03:05:41.354935 19952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:41.354162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:35449)
0929 03:05:41.354608 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:41.354698 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260929 03:05:41.805352 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:41.804555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:37569)
0929 03:05:41.805048 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:41.805145 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260929 03:05:42.250598 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:42.249810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:43237)
0929 03:05:42.250321 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:42.250417 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":243,"threads_started":1}
I20260929 03:05:42.315551 19955 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:42.315752 19955 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:42.316414 19955 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
W20260929 03:05:42.317332 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
W20260929 03:05:42.317584 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:42.317708 19557 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:42.317970 19955 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:05:42.356913 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:42.355977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:41655)
0929 03:05:42.356535 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:42.356646 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":254,"threads_started":1}
I20260929 03:05:42.807145 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:42.806330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:34299)
0929 03:05:42.806813 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:42.806908 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260929 03:05:43.252513 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:43.251637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:51943)
0929 03:05:43.252236 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:43.252340 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260929 03:05:43.358517 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:43.357805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:51639)
0929 03:05:43.358251 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:43.358352 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260929 03:05:43.809001 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:43.808211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:38323)
0929 03:05:43.808667 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:43.808804 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260929 03:05:44.254330 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:44.253555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:40049)
0929 03:05:44.254056 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:44.254151 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260929 03:05:44.360318 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:44.359417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:50733)
0929 03:05:44.359922 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:44.360046 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260929 03:05:44.810635 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:44.809829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:34771)
0929 03:05:44.810358 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:44.810448 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":243,"threads_started":1}
I20260929 03:05:45.256253 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:45.255341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:52469)
0929 03:05:45.255947 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:45.256043 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":252,"threads_started":1}
I20260929 03:05:45.361858 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:45.361162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:45813)
0929 03:05:45.361602 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:45.361699 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260929 03:05:45.812392 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:45.811602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:56747)
0929 03:05:45.812099 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:45.812202 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260929 03:05:46.257931 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:46.257226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:34317)
0929 03:05:46.257669 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:46.257751 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260929 03:05:46.363413 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:46.362658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:37805)
0929 03:05:46.363151 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:46.363240 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":239,"threads_started":1}
I20260929 03:05:46.752620 19969 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:46.752902 19969 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:46.753537 19969 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
W20260929 03:05:46.754740 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
W20260929 03:05:46.755028 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:46.755160 19557 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:46.755457 19969 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:05:46.814457 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:46.813518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:40405)
0929 03:05:46.814131 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:46.814282 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
I20260929 03:05:47.259544 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:47.258823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:50775)
0929 03:05:47.259272 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:47.259379 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":202,"threads_started":1}
I20260929 03:05:47.365118 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:47.364365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:57221)
0929 03:05:47.364849 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:47.364950 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":252,"threads_started":1}
I20260929 03:05:47.816499 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:47.815599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:50933)
0929 03:05:47.816159 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:47.816257 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":264,"threads_started":1}
I20260929 03:05:48.261349 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:48.260555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:39009)
0929 03:05:48.261053 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:48.261162 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":275,"threads_started":1}
I20260929 03:05:48.366896 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:48.366121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:38421)
0929 03:05:48.366629 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:48.366718 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260929 03:05:48.818296 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:48.817542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:54847)
0929 03:05:48.818034 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:48.818121 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":239,"threads_started":1}
I20260929 03:05:49.263010 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:49.262233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:32859)
0929 03:05:49.262735 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:49.262837 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":241,"threads_started":1}
I20260929 03:05:49.368599 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:49.367793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:49867)
0929 03:05:49.368264 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:49.368371 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260929 03:05:49.819922 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:49.819213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:46181)
0929 03:05:49.819654 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:49.819738 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260929 03:05:50.264686 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:50.263919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:51107)
0929 03:05:50.264419 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:50.264511 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
I20260929 03:05:50.370246 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:50.369505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:57531)
0929 03:05:50.369939 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:50.370032 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260929 03:05:50.821655 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:50.820927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:56721)
0929 03:05:50.821384 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:50.821483 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260929 03:05:51.266265 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:51.265552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:34969)
0929 03:05:51.266005 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:51.266101 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":215,"threads_started":1}
I20260929 03:05:51.371785 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:51.371079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:56713)
0929 03:05:51.371513 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:51.371603 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260929 03:05:51.823415 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:51.822608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:54911)
0929 03:05:51.823094 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:51.823218 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":228,"threads_started":1}
I20260929 03:05:52.267871 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:52.267125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:56727)
0929 03:05:52.267583 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:52.267669 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260929 03:05:52.373620 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:52.372906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.129:50741)
0929 03:05:52.373355 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:52.373444 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":203,"threads_started":1}
I20260929 03:05:52.825598 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:52.824735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.130:60703)
0929 03:05:52.825320 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:52.825430 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":258,"threads_started":1}
I20260929 03:05:52.844569 19989 raft_consensus.cc:482] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:52.844801 19989 raft_consensus.cc:504] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309f1c09ef734c1080e50511298ac994" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 34723 } } peers { permanent_uuid: "01319fab600141168ae2d62dbdf7b8cb" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 35907 } } peers { permanent_uuid: "39229a97e8fc43f2bf387d5676800ea2" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 33529 } }
I20260929 03:05:52.845520 19989 leader_election.cc:288] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907), 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529)
W20260929 03:05:52.846547 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01319fab600141168ae2d62dbdf7b8cb (127.2.205.130:35907): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
W20260929 03:05:52.846760 19557 leader_election.cc:341] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39229a97e8fc43f2bf387d5676800ea2 (127.2.205.131:33529): Not found: Tablet not found: 09447ef54167460fa5c1ec4b337cae2c
I20260929 03:05:52.846871 19557 leader_election.cc:302] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309f1c09ef734c1080e50511298ac994; no voters: 01319fab600141168ae2d62dbdf7b8cb, 39229a97e8fc43f2bf387d5676800ea2
I20260929 03:05:52.847118 19989 raft_consensus.cc:2726] T 09447ef54167460fa5c1ec4b337cae2c P 309f1c09ef734c1080e50511298ac994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:05:53.251466 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 19501
I20260929 03:05:53.269727 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:53.268955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:41047 (local address 127.2.205.131:57835)
0929 03:05:53.269448 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:53.269558 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.190:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260929 03:05:53.270228 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:53.252884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.2.205.129:34723)
0929 03:05:53.253460 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:53.253468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:53.253495 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:05:53.253559 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:53.253565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:53.253790 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:53.253961 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:53.255424 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.256434 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.257787 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.258433 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.258542 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:53.264548 (+ 6006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:53.264561 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:53.264576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:53.264617 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:53.267343 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.268053 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.268060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.268063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.268220 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.268585 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.268588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.268590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.269010 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:53.269306 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:53.269630 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:05:53.269903 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":248,"threads_started":1}
I20260929 03:05:53.271512 19667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:53.361981 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 19501
I20260929 03:05:53.375945 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 19675
I20260929 03:05:53.388908 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:53.376644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.2.205.130:35907)
0929 03:05:53.377182 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:53.377186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:53.377203 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:05:53.377257 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:53.377260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:53.377322 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:53.377412 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:53.378733 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.379575 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.380922 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.381337 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.381428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:53.383914 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:53.383919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:53.383922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:53.383955 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:53.386390 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.387030 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.387035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.387037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.387174 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.387466 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.387469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.387471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.387742 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:53.387945 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:53.388238 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:05:53.388459 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260929 03:05:53.389621 19798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:53.473905 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 19675
I20260929 03:05:53.488965 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 19806
I20260929 03:05:53.502856 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:53.489870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.2.205.131:33529)
0929 03:05:53.490428 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:53.490433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:53.490450 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:05:53.490504 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:53.490507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:53.490575 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:53.490665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:53.492063 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.492928 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.494192 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:53.494641 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:53.494749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:53.497387 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:53.497393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:53.497397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:53.497434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:53.500171 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.501015 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.501019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.501021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.501176 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:53.501546 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:53.501551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:53.501553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:53.501843 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:53.502066 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:53.502282 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:05:53.502477 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":293,"threads_started":1}
I20260929 03:05:53.503561 19929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:53.587401 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 19806
2026-09-29T03:05:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63933 ms)
[----------] 30 tests from SecurityITest (293981 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:05:53 dist-test-slave-530n krb5kdc[19999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:05:53 dist-test-slave-530n krb5kdc[19999](info): set up 2 sockets
Sep 29 03:05:53 dist-test-slave-530n krb5kdc[19999](info): commencing operation
krb5kdc: starting...
W20260929 03:05:54.609198 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:05:54 dist-test-slave-530n krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:05:54.848392 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40137
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:55.180353 20012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:55.180681 20012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:55.180826 20012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:55.188362 20012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:05:55.188496 20012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:55.188546 20012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:55.188629 20012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:05:55.188691 20012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:05:55.199299 20012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:40137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:55.200726 20012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:55.202153 20012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:55.214623 20017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:55.243221 20018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:55.243259 20020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:55.243307 20012 server_base.cc:1034] running on GCE node
I20260929 03:05:55.244326 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:55.245973 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:55.247207 20012 hybrid_clock.cc:648] HybridClock initialized: now 1790651155247160 us; error 36 us; skew 500 ppm
I20260929 03:05:55.247563 20012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:55 dist-test-slave-530n krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:55.250674 20012 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:05:55.252215 20012 webserver.cc:460] Webserver started at http://127.2.205.190:45161/ using document root <none> and password file <none>
I20260929 03:05:55.252938 20012 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:55.253059 20012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:55.253352 20012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:55.256338 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "5ba32f3696914d1588179adf1ce2cf65"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "4a1cc53ca391e148b35ea79b30599e54"
server_key_iv: "e9fc6aaf6dace301c930654b1b7ddb0e"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.257121 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "5ba32f3696914d1588179adf1ce2cf65"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "4a1cc53ca391e148b35ea79b30599e54"
server_key_iv: "e9fc6aaf6dace301c930654b1b7ddb0e"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.262265 20012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:05:55.265661 20027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:55.266909 20012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:05:55.267063 20012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "5ba32f3696914d1588179adf1ce2cf65"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "4a1cc53ca391e148b35ea79b30599e54"
server_key_iv: "e9fc6aaf6dace301c930654b1b7ddb0e"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.267232 20012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:55.288298 20012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:55.293076 20012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:55.293349 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:55.311565 20012 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:40137
I20260929 03:05:55.311604 20078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:40137 every 8 connection(s)
I20260929 03:05:55.313047 20012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:05:55.317164 20079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:55.321704 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20012
I20260929 03:05:55.321911 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:05:55.322556 2870 external_mini_cluster.cc:1428] Setting key 6036ef1689bbcb6299748db11a73b47e
I20260929 03:05:55.330174 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: Bootstrap starting.
I20260929 03:05:55.334128 20079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:55.335263 20079 log.cc:824] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:55.338446 20079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: No bootstrap required, opened a new log
Sep 29 03:05:55 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:55.349185 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:55.324669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.2.205.190:40137)
0929 03:05:55.325621 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:55.325639 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:55.325698 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:05:55.328252 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:55.328289 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:55.328929 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:55.329428 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:55.330923 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.332996 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.334448 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.335196 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.335386 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:55.339780 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:55.339792 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:55.339814 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:55.339865 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:55.344082 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:55.345041 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:55.345053 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:55.345062 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:55.345295 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:55.345668 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:55.345673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:55.345676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:55.346473 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:55.346784 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:55.347227 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 03:05:55.347726 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":375,"threads_started":1}
I20260929 03:05:55.348943 20079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } }
I20260929 03:05:55.349414 20079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:55.349565 20079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba32f3696914d1588179adf1ce2cf65, State: Initialized, Role: FOLLOWER
I20260929 03:05:55.350147 20079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } }
I20260929 03:05:55.350383 20079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:05:55.350510 20079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:05:55.350682 20079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:55.352996 20079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } }
I20260929 03:05:55.353471 20079 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba32f3696914d1588179adf1ce2cf65; no voters:
I20260929 03:05:55.354020 20079 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:05:55.354157 20084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:05:55.355060 20084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba32f3696914d1588179adf1ce2cf65, State: Running, Role: LEADER
I20260929 03:05:55.355575 20084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } }
I20260929 03:05:55.356006 20079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:05:55.358350 20085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba32f3696914d1588179adf1ce2cf65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } } }
I20260929 03:05:55.358425 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba32f3696914d1588179adf1ce2cf65. Latest consensus state: current_term: 1 leader_uuid: "5ba32f3696914d1588179adf1ce2cf65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba32f3696914d1588179adf1ce2cf65" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 40137 } } }
I20260929 03:05:55.358721 20085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:55.358758 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65 [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:55.359568 20091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:05:55.364302 20091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:05:55.372640 20091 catalog_manager.cc:1347] Generated new cluster ID: 74f39182d7854f618664d3c6268dbe1d
I20260929 03:05:55.372802 20091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:05:55.388515 20091 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:05:55.389914 20091 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:05:55.403744 20091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: Generated new TSK 0
I20260929 03:05:55.404671 20091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:05:55.464087 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:55.802026 20105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:55.802330 20105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:55.802423 20105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:55.810140 20105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:55.810266 20105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:55.810397 20105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:05:55.820544 20105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:55.821918 20105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:55.823248 20105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:55.836009 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:55.836184 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:55.836545 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:55.837450 20105 server_base.cc:1034] running on GCE node
I20260929 03:05:55.838102 20105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:55.839133 20105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:55.840354 20105 hybrid_clock.cc:648] HybridClock initialized: now 1790651155840294 us; error 49 us; skew 500 ppm
I20260929 03:05:55.840699 20105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:55 dist-test-slave-530n krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:55.844077 20105 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:05:55.845851 20105 webserver.cc:460] Webserver started at http://127.2.205.129:32975/ using document root <none> and password file <none>
I20260929 03:05:55.846565 20105 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:55.846674 20105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:55.846997 20105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:55.850104 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "1252f89344b14a2184dd6ba21a47ef02"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "2043693f9e1664996e115a312a7a7617"
server_key_iv: "f278bbdb8651a6967c0295bcf5062be7"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.850896 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "1252f89344b14a2184dd6ba21a47ef02"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "2043693f9e1664996e115a312a7a7617"
server_key_iv: "f278bbdb8651a6967c0295bcf5062be7"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.856597 20105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:05:55.860157 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:55.861518 20105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:05:55.861675 20105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "1252f89344b14a2184dd6ba21a47ef02"
format_stamp: "Formatted at 2026-09-29 03:05:55 on dist-test-slave-530n"
server_key: "2043693f9e1664996e115a312a7a7617"
server_key_iv: "f278bbdb8651a6967c0295bcf5062be7"
server_key_version: "encryptionkey@0"
I20260929 03:05:55.861873 20105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:55.893603 20105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:55.897804 20105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:55.898041 20105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:55.898981 20105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:55.900264 20105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:55.900333 20105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:55.900424 20105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:55.900472 20105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:55.931340 20105 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:46409
I20260929 03:05:55.931411 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:46409 every 8 connection(s)
I20260929 03:05:55.932701 20105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:05:55.937650 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20105
I20260929 03:05:55.937855 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:05:55.938627 2870 external_mini_cluster.cc:1428] Setting key 0a694315b43c4eb3443b701b00505c3d
Sep 29 03:05:55 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:55.954042 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:55.935224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:53991 (local address 127.2.205.190:40137)
0929 03:05:55.935663 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:55.935668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:55.936632 (+ 964us) server_negotiation.cc:407] Connection header received
0929 03:05:55.939009 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:55.939014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:55.939090 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:55.939186 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:55.940963 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.941723 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.943890 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.944312 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.944423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:55.948390 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:55.948395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:55.948398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:55.948433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:55.951104 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:55.951808 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:55.951813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:55.951815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:55.951935 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:55.952343 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:55.952347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:55.952349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:55.952620 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:55.952782 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:55.953464 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 03:05:55.953691 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":209,"threads_started":1}
I20260929 03:05:55.955008 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:55.935788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40137 (local address 127.2.205.129:53991)
0929 03:05:55.936464 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:55.936502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:55.938698 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:55.939422 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:55.939430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:55.940043 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:55.940769 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:55.940796 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.941890 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.941895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:55.943730 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:55.943744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:55.944523 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:55.944529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:55.944846 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:55.945492 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:55.945519 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:55.948154 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:55.951236 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:55.951242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:55.951256 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:55.951679 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:55.952043 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:55.952049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:55.952051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:55.952228 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:55.952739 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:55.952766 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:55.953103 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 03:05:55.953338 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 03:05:55.953868 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260929 03:05:55.956634 20233 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40137
I20260929 03:05:55.956967 20233 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:55.957744 20233 heartbeater.cc:511] Master 127.2.205.190:40137 requested a full tablet report, sending...
I20260929 03:05:55.960167 20044 ts_manager.cc:194] Registered new tserver with Master: 1252f89344b14a2184dd6ba21a47ef02 (127.2.205.129:46409)
I20260929 03:05:55.962232 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:53991
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:05:55.993953 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:56.335901 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:56.336190 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:56.336326 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:56.343732 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:56.343870 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:56.344038 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:05:56.354002 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:56.355332 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:56.356715 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:56.369272 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:56.369278 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:56.369302 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:56.370220 20239 server_base.cc:1034] running on GCE node
I20260929 03:05:56.370769 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:56.371610 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:56.372821 20239 hybrid_clock.cc:648] HybridClock initialized: now 1790651156372776 us; error 36 us; skew 500 ppm
I20260929 03:05:56.373096 20239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:56 dist-test-slave-530n krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:56.375964 20239 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:05:56.377367 20239 webserver.cc:460] Webserver started at http://127.2.205.130:41657/ using document root <none> and password file <none>
I20260929 03:05:56.378003 20239 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:56.378118 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:56.378402 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:56.381263 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "6059cba42123447e80f5d56c94a46d7f"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "a73f0253fe1c59eb833c6af21c47cf4c"
server_key_iv: "ff10be6ec6be85bbf35a3c32966937ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.382037 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "6059cba42123447e80f5d56c94a46d7f"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "a73f0253fe1c59eb833c6af21c47cf4c"
server_key_iv: "ff10be6ec6be85bbf35a3c32966937ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.387143 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:05:56.390573 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.391955 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:05:56.392115 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "6059cba42123447e80f5d56c94a46d7f"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "a73f0253fe1c59eb833c6af21c47cf4c"
server_key_iv: "ff10be6ec6be85bbf35a3c32966937ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.392303 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:56.415256 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:56.419960 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:56.420265 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:56.421386 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:56.422806 20239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:56.422891 20239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.423070 20239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:56.423197 20239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.454224 20239 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:42245
I20260929 03:05:56.454280 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:42245 every 8 connection(s)
I20260929 03:05:56.455526 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:05:56.457561 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20239
I20260929 03:05:56.457707 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:05:56.458199 2870 external_mini_cluster.cc:1428] Setting key 8d152879d43673c1a91640d8366de566
Sep 29 03:05:56 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:56.475572 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:56.458080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:33199 (local address 127.2.205.190:40137)
0929 03:05:56.461107 (+ 3027us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:56.461113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:56.461133 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:05:56.461795 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:56.461800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:56.461876 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:56.461962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:56.463410 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.464142 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.465870 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.466346 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.466453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:56.470160 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:56.470165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:56.470168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:56.470200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:56.472690 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:56.473396 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:56.473400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:56.473403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:56.473523 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:56.473920 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:56.473924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:56.473926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:56.474187 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:56.474315 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:56.474970 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 03:05:56.475185 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2897,"thread_start_us":2802,"threads_started":1}
I20260929 03:05:56.476630 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:56.458718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40137 (local address 127.2.205.130:33199)
0929 03:05:56.459593 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:56.459640 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:56.461552 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:56.462200 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:56.462208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:56.462697 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:56.463255 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:56.463272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.464275 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.464279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:56.465747 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:56.465759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.466563 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.466570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:56.466844 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:56.467497 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:56.467530 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:56.469938 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:56.472843 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:56.472852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:56.472868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:56.473279 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:56.473622 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:56.473627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:56.473631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:56.473813 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:56.474304 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:56.474311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:56.474630 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 03:05:56.474856 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 03:05:56.475376 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":330,"threads_started":1}
I20260929 03:05:56.478225 20367 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40137
I20260929 03:05:56.478572 20367 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:56.479653 20367 heartbeater.cc:511] Master 127.2.205.190:40137 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:05:56.481410 20044 ts_manager.cc:194] Registered new tserver with Master: 6059cba42123447e80f5d56c94a46d7f (127.2.205.130:42245)
I20260929 03:05:56.482255 20044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:33199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:05:56.510308 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:05:56.837198 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:56.837502 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:56.837628 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:56.845024 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:56.845150 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:56.845309 20373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:05:56.855216 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:56.856577 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:56.857967 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:56.871017 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:56.870988 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:56.870962 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:56.871706 20373 server_base.cc:1034] running on GCE node
I20260929 03:05:56.872406 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:56.873504 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:56.874724 20373 hybrid_clock.cc:648] HybridClock initialized: now 1790651156874660 us; error 50 us; skew 500 ppm
I20260929 03:05:56.875054 20373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:05:56 dist-test-slave-530n krb5kdc[19999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:05:56.878252 20373 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:05:56.879758 20373 webserver.cc:460] Webserver started at http://127.2.205.131:40881/ using document root <none> and password file <none>
I20260929 03:05:56.880467 20373 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:56.880599 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:56.880947 20373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:56.883972 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "3e627fd7669344bebfad13c3af996b27"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "1767485040ae307b1ba5bdc9cf67988f"
server_key_iv: "163852b6d457a34ee33b9266e49c1988"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.884828 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "3e627fd7669344bebfad13c3af996b27"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "1767485040ae307b1ba5bdc9cf67988f"
server_key_iv: "163852b6d457a34ee33b9266e49c1988"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.890231 20373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 03:05:56.893741 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.895139 20373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:05:56.895361 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "3e627fd7669344bebfad13c3af996b27"
format_stamp: "Formatted at 2026-09-29 03:05:56 on dist-test-slave-530n"
server_key: "1767485040ae307b1ba5bdc9cf67988f"
server_key_iv: "163852b6d457a34ee33b9266e49c1988"
server_key_version: "encryptionkey@0"
I20260929 03:05:56.895593 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:56.922609 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:56.927419 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:56.927685 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:56.928640 20373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:56.930101 20373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:56.930182 20373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.930269 20373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:56.930311 20373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:56.961212 20373 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:35827
I20260929 03:05:56.961261 20500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:35827 every 8 connection(s)
I20260929 03:05:56.962451 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:05:56.963883 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20373
I20260929 03:05:56.964036 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:05:56.964506 2870 external_mini_cluster.cc:1428] Setting key 3d4d627a6a841a51318f97e3e54db2a5
I20260929 03:05:56.965497 20233 heartbeater.cc:503] Master 127.2.205.190:40137 was elected leader, sending a full tablet report...
Sep 29 03:05:56 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:05:56.982955 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:56.964980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:41607 (local address 127.2.205.190:40137)
0929 03:05:56.965193 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:56.965200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:56.966628 (+ 1428us) server_negotiation.cc:407] Connection header received
0929 03:05:56.968922 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:56.968927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:56.969000 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:56.969101 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:56.970659 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.971471 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.973318 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.973715 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.973809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:56.977091 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:56.977096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:56.977099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:56.977130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:56.979943 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:56.980832 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:56.980836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:56.980838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:56.980957 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:56.981355 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:56.981358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:56.981361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:56.981646 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:56.981764 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:56.982531 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 03:05:56.982711 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:05:56.983960 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:56.965546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:40137 (local address 127.2.205.131:41607)
0929 03:05:56.966471 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:56.966529 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:56.968648 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:56.969351 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:56.969361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:56.969847 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:56.970502 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:56.970522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.971614 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.971618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:56.973184 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:56.973199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:56.973895 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:56.973899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:56.974099 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:56.974713 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:05:56.974734 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:56.976884 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:56.980071 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:56.980081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:56.980098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:56.980700 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:56.981065 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:05:56.981068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:05:56.981070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:05:56.981251 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:05:56.981777 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:56.981784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:56.982147 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 03:05:56.982389 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 03:05:56.982793 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":387,"threads_started":1}
I20260929 03:05:56.985373 20501 heartbeater.cc:347] Connected to a master server at 127.2.205.190:40137
I20260929 03:05:56.985673 20501 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:56.986411 20501 heartbeater.cc:511] Master 127.2.205.190:40137 requested a full tablet report, sending...
I20260929 03:05:56.987843 20043 ts_manager.cc:194] Registered new tserver with Master: 3e627fd7669344bebfad13c3af996b27 (127.2.205.131:35827)
I20260929 03:05:56.988727 20043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:41607
I20260929 03:05:56.992834 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:05:57.010284 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:56.997443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.2.205.190:40137)
0929 03:05:56.997687 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:56.997694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:56.997948 (+ 254us) server_negotiation.cc:407] Connection header received
0929 03:05:56.998115 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:56.998119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:56.998190 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:56.998273 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:56.999774 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.000549 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.001904 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.002320 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.002414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.004875 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:57.004880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:57.004884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:57.004916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:57.007470 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.008208 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.008212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.008214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.008371 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.008777 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.008781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.008783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.009144 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:57.009362 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.009777 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.009957 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:05:57.014478 20043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:05:57.021325 20043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:05:57.057519 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.033877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:35827 (local address 127.0.0.1:46720)
0929 03:05:57.047839 (+ 13962us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:57.047905 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:57.048151 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:57.049629 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:57.049637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:57.049671 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:57.050138 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.050152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.052084 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.052089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.054693 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.054709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.056495 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.056500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.056725 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.056811 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 03:05:57.056937 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:05:57.057134 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13630,"thread_start_us":13476,"threads_started":1}
I20260929 03:05:57.057927 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.034639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.2.205.129:46409)
0929 03:05:57.046914 (+ 12275us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.046926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.048227 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 03:05:57.049939 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.049948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.050221 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.050373 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:57.051231 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.052717 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.055372 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.056776 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.056910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.057138 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.057370 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.057527 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12047,"thread_start_us":12306,"threads_started":1}
I20260929 03:05:57.058148 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.033992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:46409 (local address 127.0.0.1:47166)
0929 03:05:57.048106 (+ 14114us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:57.048122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:57.049706 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:57.050647 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:57.050651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:57.050677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:57.051058 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.051071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.052909 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.052914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.055183 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.055199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.056970 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.056976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.057183 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.057233 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:05:57.057320 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:05:57.057445 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13983,"mutex_wait_us":41,"spinlock_wait_cycles":15104,"thread_start_us":14347,"threads_started":1}
I20260929 03:05:57.058388 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.034400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.2.205.131:35827)
0929 03:05:57.048725 (+ 14325us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.048733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.048784 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:05:57.048888 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.048895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.049175 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.049363 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:57.050366 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.051924 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.054857 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.056203 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.056324 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.056540 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.057909 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.058072 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14161,"thread_start_us":13989,"threads_started":1}
I20260929 03:05:57.061913 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.049183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:42245 (local address 127.0.0.1:56538)
0929 03:05:57.054321 (+ 5138us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:57.054337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:57.054450 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:57.054913 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:57.054918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:57.054943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:57.055332 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.055343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.057109 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.057114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.059241 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.059255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.061140 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.061146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.061305 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.061336 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:05:57.061428 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:05:57.061544 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5006,"mutex_wait_us":4,"thread_start_us":4736,"threads_started":1}
I20260929 03:05:57.061929 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.048815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.2.205.130:42245)
0929 03:05:57.053830 (+ 5015us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.053839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.054427 (+ 588us) server_negotiation.cc:407] Connection header received
0929 03:05:57.054594 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.054600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.054747 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.054937 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:57.055493 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.056964 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.059380 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.060696 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.060843 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.061013 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.061497 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.061658 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4874,"thread_start_us":4778,"threads_started":1}
I20260929 03:05:57.061884 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee2af1eec084d81b819b2428055d660 (DEFAULT_TABLE table=test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:57.061970 20436 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee2af1eec084d81b819b2428055d660 (DEFAULT_TABLE table=test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:57.063217 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee2af1eec084d81b819b2428055d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:57.063237 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee2af1eec084d81b819b2428055d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:57.065181 20302 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee2af1eec084d81b819b2428055d660 (DEFAULT_TABLE table=test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:57.066431 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee2af1eec084d81b819b2428055d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:57.074409 20519 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Bootstrap starting.
I20260929 03:05:57.077026 20520 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Bootstrap starting.
I20260929 03:05:57.078264 20519 tablet_bootstrap.cc:654] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:57.079559 20519 log.cc:824] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:57.080257 20521 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Bootstrap starting.
I20260929 03:05:57.080827 20520 tablet_bootstrap.cc:654] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:57.082188 20520 log.cc:824] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:57.083379 20521 tablet_bootstrap.cc:654] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:57.084285 20521 log.cc:824] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:57.087293 20521 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: No bootstrap required, opened a new log
I20260929 03:05:57.087558 20519 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: No bootstrap required, opened a new log
I20260929 03:05:57.087570 20521 ts_tablet_manager.cc:1397] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260929 03:05:57.087604 20520 tablet_bootstrap.cc:492] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: No bootstrap required, opened a new log
I20260929 03:05:57.087841 20520 ts_tablet_manager.cc:1397] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260929 03:05:57.087845 20519 ts_tablet_manager.cc:1397] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260929 03:05:57.095412 20521 raft_consensus.cc:348] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.095947 20521 raft_consensus.cc:374] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:57.096045 20521 raft_consensus.cc:729] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6059cba42123447e80f5d56c94a46d7f, State: Initialized, Role: FOLLOWER
I20260929 03:05:57.096724 20521 consensus_queue.cc:260] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.097971 20367 heartbeater.cc:503] Master 127.2.205.190:40137 was elected leader, sending a full tablet report...
I20260929 03:05:57.098719 20521 ts_tablet_manager.cc:1428] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260929 03:05:57.098568 20520 raft_consensus.cc:348] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.098575 20519 raft_consensus.cc:348] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.099032 20519 raft_consensus.cc:374] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:57.099013 20520 raft_consensus.cc:374] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:57.099128 20519 raft_consensus.cc:729] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e627fd7669344bebfad13c3af996b27, State: Initialized, Role: FOLLOWER
I20260929 03:05:57.099172 20520 raft_consensus.cc:729] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1252f89344b14a2184dd6ba21a47ef02, State: Initialized, Role: FOLLOWER
I20260929 03:05:57.099761 20520 consensus_queue.cc:260] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.099756 20519 consensus_queue.cc:260] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.101646 20520 ts_tablet_manager.cc:1428] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260929 03:05:57.103442 20519 ts_tablet_manager.cc:1428] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 03:05:57.103960 20501 heartbeater.cc:503] Master 127.2.205.190:40137 was elected leader, sending a full tablet report...
I20260929 03:05:57.133137 20525 raft_consensus.cc:482] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:57.133487 20525 raft_consensus.cc:504] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.135371 20525 leader_election.cc:288] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e627fd7669344bebfad13c3af996b27 (127.2.205.131:35827), 1252f89344b14a2184dd6ba21a47ef02 (127.2.205.129:46409)
I20260929 03:05:57.144516 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.136210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:39357 (local address 127.2.205.129:46409)
0929 03:05:57.136436 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.136442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.137016 (+ 574us) server_negotiation.cc:407] Connection header received
0929 03:05:57.137172 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.137177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.137263 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.137350 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:57.137967 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.139046 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.142018 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.143312 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.143493 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.143572 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.144132 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.144258 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:05:57.144511 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.135731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:38927 (local address 127.2.205.131:35827)
0929 03:05:57.135964 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.135970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.136735 (+ 765us) server_negotiation.cc:407] Connection header received
0929 03:05:57.136975 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.136980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.137073 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.137176 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:57.137768 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.138887 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.142010 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.143312 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.143493 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.143576 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.144111 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.144255 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 03:05:57.144769 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.135749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:46409 (local address 127.2.205.130:39357)
0929 03:05:57.136912 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:57.136929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:57.137031 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:57.137388 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:57.137392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:57.137411 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:57.137821 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.137831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.139188 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.139193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.141755 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.141782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.143638 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.143643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.143903 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.143963 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 03:05:57.144068 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:05:57.144264 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":274,"thread_start_us":827,"threads_started":1}
I20260929 03:05:57.145169 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.135633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:35827 (local address 127.2.205.130:38927)
0929 03:05:57.136596 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:57.136617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:57.136829 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:57.137204 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:57.137207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:57.137227 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:57.137623 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.137635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.139070 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.139074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.141756 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:57.141782 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.143618 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.143630 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:57.143900 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.143965 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 03:05:57.144074 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:05:57.144264 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":196,"thread_start_us":605,"threads_started":1}
I20260929 03:05:57.145844 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee2af1eec084d81b819b2428055d660" candidate_uuid: "6059cba42123447e80f5d56c94a46d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1252f89344b14a2184dd6ba21a47ef02" is_pre_election: true
I20260929 03:05:57.146142 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee2af1eec084d81b819b2428055d660" candidate_uuid: "6059cba42123447e80f5d56c94a46d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e627fd7669344bebfad13c3af996b27" is_pre_election: true
I20260929 03:05:57.146347 20188 raft_consensus.cc:2454] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6059cba42123447e80f5d56c94a46d7f in term 0.
I20260929 03:05:57.146600 20456 raft_consensus.cc:2454] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6059cba42123447e80f5d56c94a46d7f in term 0.
I20260929 03:05:57.147060 20257 leader_election.cc:302] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1252f89344b14a2184dd6ba21a47ef02, 6059cba42123447e80f5d56c94a46d7f; no voters:
I20260929 03:05:57.147429 20525 raft_consensus.cc:2781] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:05:57.147575 20525 raft_consensus.cc:482] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:05:57.147673 20525 raft_consensus.cc:3037] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:57.149887 20525 raft_consensus.cc:504] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.150514 20525 leader_election.cc:288] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [CANDIDATE]: Term 1 election: Requested vote from peers 3e627fd7669344bebfad13c3af996b27 (127.2.205.131:35827), 1252f89344b14a2184dd6ba21a47ef02 (127.2.205.129:46409)
I20260929 03:05:57.150939 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee2af1eec084d81b819b2428055d660" candidate_uuid: "6059cba42123447e80f5d56c94a46d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e627fd7669344bebfad13c3af996b27"
I20260929 03:05:57.150986 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee2af1eec084d81b819b2428055d660" candidate_uuid: "6059cba42123447e80f5d56c94a46d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1252f89344b14a2184dd6ba21a47ef02"
I20260929 03:05:57.151113 20456 raft_consensus.cc:3037] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:57.151186 20188 raft_consensus.cc:3037] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:57.153584 20456 raft_consensus.cc:2454] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6059cba42123447e80f5d56c94a46d7f in term 1.
I20260929 03:05:57.153687 20188 raft_consensus.cc:2454] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6059cba42123447e80f5d56c94a46d7f in term 1.
I20260929 03:05:57.153934 20255 leader_election.cc:302] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e627fd7669344bebfad13c3af996b27, 6059cba42123447e80f5d56c94a46d7f; no voters:
I20260929 03:05:57.154258 20525 raft_consensus.cc:2781] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:05:57.154675 20525 raft_consensus.cc:686] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 1 LEADER]: Becoming Leader. State: Replica: 6059cba42123447e80f5d56c94a46d7f, State: Running, Role: LEADER
I20260929 03:05:57.155151 20525 consensus_queue.cc:237] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } }
I20260929 03:05:57.159396 20044 catalog_manager.cc:5495] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6059cba42123447e80f5d56c94a46d7f (127.2.205.130). New cstate: current_term: 1 leader_uuid: "6059cba42123447e80f5d56c94a46d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6059cba42123447e80f5d56c94a46d7f" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 42245 } health_report { overall_health: HEALTHY } } }
W20260929 03:05:57.185168 20234 tablet.cc:2369] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:05:57.188874 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.183473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.2.205.130:42245)
0929 03:05:57.183690 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.183695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.183710 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:05:57.183797 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.183801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.183878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.184012 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:05:57.184422 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.185358 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.186854 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.187275 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.187370 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.187797 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:05:57.188314 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:05:57.188422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.188540 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.188619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:05:57.197347 20456 raft_consensus.cc:1261] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 1 FOLLOWER]: Refusing update from remote peer 6059cba42123447e80f5d56c94a46d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:57.197353 20188 raft_consensus.cc:1261] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 1 FOLLOWER]: Refusing update from remote peer 6059cba42123447e80f5d56c94a46d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:57.198359 20530 consensus_queue.cc:1035] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e627fd7669344bebfad13c3af996b27" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 35827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:05:57.198729 20525 consensus_queue.cc:1035] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1252f89344b14a2184dd6ba21a47ef02" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:05:57.208389 20368 tablet.cc:2369] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:05:57.211443 20533 mvcc.cc:205] Tried to move back new op lower bound from 7334507139869544448 to 7334507139711340544. Current Snapshot: MvccSnapshot[applied={T|T < 7334507139869544448}]
I20260929 03:05:57.224916 20044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:57.225221 20044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:57.229171 20044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ba32f3696914d1588179adf1ce2cf65: Sending DeleteTablet for 3 replicas of tablet 5ee2af1eec084d81b819b2428055d660
I20260929 03:05:57.230463 20168 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee2af1eec084d81b819b2428055d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:57 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:47166
I20260929 03:05:57.230552 20436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee2af1eec084d81b819b2428055d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:57 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:46720
I20260929 03:05:57.235078 20302 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee2af1eec084d81b819b2428055d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:57 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:56538
I20260929 03:05:57.236585 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20105
I20260929 03:05:57.247588 20542 tablet_replica.cc:330] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: stopping tablet replica
I20260929 03:05:57.248288 20542 raft_consensus.cc:2229] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:05:57.249034 20542 raft_consensus.cc:2258] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:57.251518 20542 ts_tablet_manager.cc:1905] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:05:57.252131 20543 tablet_replica.cc:330] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: stopping tablet replica
I20260929 03:05:57.252602 20543 raft_consensus.cc:2229] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:05:57.254134 20544 tablet_replica.cc:330] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: stopping tablet replica
I20260929 03:05:57.254578 20544 raft_consensus.cc:2229] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:05:57.255275 20544 raft_consensus.cc:2258] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:57.256160 20543 raft_consensus.cc:2258] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:57.258426 20544 ts_tablet_manager.cc:1905] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:05:57.258765 20543 ts_tablet_manager.cc:1905] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 03:05:57 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:05:57.264427 20542 ts_tablet_manager.cc:1918] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:05:57.264603 20542 log.cc:1196] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/5ee2af1eec084d81b819b2428055d660
I20260929 03:05:57.265190 20542 ts_tablet_manager.cc:1939] T 5ee2af1eec084d81b819b2428055d660 P 1252f89344b14a2184dd6ba21a47ef02: Deleting consensus metadata
I20260929 03:05:57.265304 20543 ts_tablet_manager.cc:1918] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:05:57.265491 20543 log.cc:1196] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/5ee2af1eec084d81b819b2428055d660
I20260929 03:05:57.266027 20543 ts_tablet_manager.cc:1939] T 5ee2af1eec084d81b819b2428055d660 P 3e627fd7669344bebfad13c3af996b27: Deleting consensus metadata
I20260929 03:05:57.267640 20028 catalog_manager.cc:4841] TS 3e627fd7669344bebfad13c3af996b27 (127.2.205.131:35827): tablet 5ee2af1eec084d81b819b2428055d660 (table test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]) successfully deleted
I20260929 03:05:57.267964 20030 catalog_manager.cc:4841] TS 1252f89344b14a2184dd6ba21a47ef02 (127.2.205.129:46409): tablet 5ee2af1eec084d81b819b2428055d660 (table test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]) successfully deleted
I20260929 03:05:57.269104 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.238931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.2.205.129:46409)
0929 03:05:57.239156 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.239163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.248176 (+ 9013us) server_negotiation.cc:407] Connection header received
0929 03:05:57.248734 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.248739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.248853 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.248938 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:57.250823 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.251874 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.254397 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.255075 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.255180 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.261303 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:57.261314 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:57.261327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:57.261363 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:57.265287 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.266440 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.266450 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.266453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.266650 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.267150 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.267155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.267159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.267576 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:57.267872 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.268481 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.268717 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:05:57.269788 20228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:57.270385 20544 ts_tablet_manager.cc:1918] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:05:57.270545 20544 log.cc:1196] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/5ee2af1eec084d81b819b2428055d660
I20260929 03:05:57.271188 20544 ts_tablet_manager.cc:1939] T 5ee2af1eec084d81b819b2428055d660 P 6059cba42123447e80f5d56c94a46d7f: Deleting consensus metadata
I20260929 03:05:57.273252 20028 catalog_manager.cc:4841] TS 6059cba42123447e80f5d56c94a46d7f (127.2.205.130:42245): tablet 5ee2af1eec084d81b819b2428055d660 (table test-table [id=483ecd6eb2154fe1b3fb2088a1e940f4]) successfully deleted
I20260929 03:05:57.359714 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20105
I20260929 03:05:57.375339 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20239
Sep 29 03:05:57 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:05:57.391193 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.376024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.2.205.130:42245)
0929 03:05:57.376211 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.376216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.376309 (+ 93us) server_negotiation.cc:407] Connection header received
0929 03:05:57.376489 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.376493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.376566 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.376652 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:57.377974 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.378844 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.380140 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.380565 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.380680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.385166 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:57.385182 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:57.385203 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:57.385244 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:57.388255 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.389023 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.389031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.389033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.389189 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.389601 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.389605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.389607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.389967 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:57.390290 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.390597 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.390823 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:05:57.391916 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:57.504292 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20239
I20260929 03:05:57.519199 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20373
Sep 29 03:05:57 dist-test-slave-530n krb5kdc[19999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:05:57.534489 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:57.519928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.2.205.131:35827)
0929 03:05:57.520139 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:57.520145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:57.520166 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:05:57.520292 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:57.520296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:57.520370 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:57.520455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:57.521816 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.522538 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.523795 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:57.524203 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:57.524311 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:57.528512 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:57.528529 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:57.528547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:05:57.528585 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:57.531402 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.532217 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.532226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.532228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.532410 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:05:57.532865 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:05:57.532870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:05:57.532872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:05:57.533275 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:57.533563 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:57.533917 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 03:05:57.534168 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:05:57.535190 20496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:57.624540 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20373
I20260929 03:05:57.637750 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20012
I20260929 03:05:57.638901 20074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:05:57.750851 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20012
2026-09-29T03:05:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:05:57Z Disabled control of system clock
I20260929 03:05:57.824352 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37761
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:05:58.152173 20553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:58.152459 20553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:58.152537 20553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:58.159567 20553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:05:58.159665 20553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:58.159708 20553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:58.159763 20553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:05:58.159806 20553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:05:58.168638 20553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:37761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:58.169850 20553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:58.171495 20553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:58.182746 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:58.182816 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:58.182789 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:58.183642 20553 server_base.cc:1034] running on GCE node
I20260929 03:05:58.184453 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:58.186053 20553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:58.187283 20553 hybrid_clock.cc:648] HybridClock initialized: now 1790651158187236 us; error 47 us; skew 500 ppm
I20260929 03:05:58.187629 20553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:05:58.189985 20553 webserver.cc:460] Webserver started at http://127.2.205.190:37487/ using document root <none> and password file <none>
I20260929 03:05:58.190667 20553 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:58.190771 20553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:58.191056 20553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:58.194284 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "b5fa4e324e514a4e8df0cae46265af02"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "cf4c03d3d01c052241bb6172892c971c"
server_key_iv: "de85fdd7c7450af4f17457d82bf10e20"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.195119 20553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "b5fa4e324e514a4e8df0cae46265af02"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "cf4c03d3d01c052241bb6172892c971c"
server_key_iv: "de85fdd7c7450af4f17457d82bf10e20"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.200703 20553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:05:58.204351 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:58.205849 20553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:05:58.206015 20553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "b5fa4e324e514a4e8df0cae46265af02"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "cf4c03d3d01c052241bb6172892c971c"
server_key_iv: "de85fdd7c7450af4f17457d82bf10e20"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.206209 20553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:58.223404 20553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:58.224323 20553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:58.224588 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:58.242666 20553 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:37761
I20260929 03:05:58.242684 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:37761 every 8 connection(s)
I20260929 03:05:58.244131 20553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:05:58.247265 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20553
I20260929 03:05:58.247416 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:05:58.247891 2870 external_mini_cluster.cc:1428] Setting key e56629f9fa362f086b914b58a306bd36
I20260929 03:05:58.248091 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:58.262876 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: Bootstrap starting.
I20260929 03:05:58.263015 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:58.249812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.2.205.190:37761)
0929 03:05:58.250852 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:58.250866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:58.250920 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:05:58.252973 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:58.253005 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:58.253633 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:58.254086 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:58.254645 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.256564 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.257988 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.258986 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.259188 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:58.260249 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:58.260259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:58.260278 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:05:58.261011 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:58.261109 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:05:58.261134 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:58.261237 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:58.261435 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:05:58.261765 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":295,"threads_started":1}
I20260929 03:05:58.266027 20619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:58.266940 20619 log.cc:824] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:58.269609 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: No bootstrap required, opened a new log
I20260929 03:05:58.277280 20619 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } }
I20260929 03:05:58.277622 20619 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:58.277698 20619 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5fa4e324e514a4e8df0cae46265af02, State: Initialized, Role: FOLLOWER
I20260929 03:05:58.278208 20619 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } }
I20260929 03:05:58.278409 20619 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:05:58.278501 20619 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:05:58.278620 20619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:58.280987 20619 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } }
I20260929 03:05:58.281481 20619 leader_election.cc:302] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5fa4e324e514a4e8df0cae46265af02; no voters:
I20260929 03:05:58.281985 20619 leader_election.cc:288] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:05:58.282148 20624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:05:58.282795 20624 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [term 1 LEADER]: Becoming Leader. State: Replica: b5fa4e324e514a4e8df0cae46265af02, State: Running, Role: LEADER
I20260929 03:05:58.283286 20624 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } }
I20260929 03:05:58.283963 20619 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:05:58.285934 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5fa4e324e514a4e8df0cae46265af02. Latest consensus state: current_term: 1 leader_uuid: "b5fa4e324e514a4e8df0cae46265af02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } } }
I20260929 03:05:58.286048 20625 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5fa4e324e514a4e8df0cae46265af02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fa4e324e514a4e8df0cae46265af02" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37761 } } }
I20260929 03:05:58.286375 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:58.286379 20625 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02 [sys.catalog]: This master's current role is: LEADER
I20260929 03:05:58.288053 20631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:05:58.293133 20631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:05:58.302253 20631 catalog_manager.cc:1347] Generated new cluster ID: 0a01b1c9de4444939f0a62f926ea2add
I20260929 03:05:58.302366 20631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:05:58.336957 20631 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:05:58.338358 20631 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:05:58.345681 20631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: Generated new TSK 0
I20260929 03:05:58.346575 20631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:05:58.352244 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--builtin_ntp_servers=127.2.205.148:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:05:58.715859 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:58.716156 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:58.716239 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:58.723526 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:58.723641 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:58.723749 20643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:05:58.733052 20643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:58.734315 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:58.736121 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:58.749404 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:58.749372 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:58.749737 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:58.750562 20643 server_base.cc:1034] running on GCE node
I20260929 03:05:58.751257 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:58.752332 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:58.753579 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790651158753522 us; error 48 us; skew 500 ppm
I20260929 03:05:58.753875 20643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:05:58.759773 20643 webserver.cc:460] Webserver started at http://127.2.205.129:38349/ using document root <none> and password file <none>
I20260929 03:05:58.760445 20643 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:58.760533 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:58.760864 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:58.764490 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "08d6c8dd9349402f8fd9253bb9a5d240"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "a8ad38c9134b454f78baa2c9c0491501"
server_key_iv: "38dd53c9aa3baa1d1fba9616c6c9b757"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.765256 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "08d6c8dd9349402f8fd9253bb9a5d240"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "a8ad38c9134b454f78baa2c9c0491501"
server_key_iv: "38dd53c9aa3baa1d1fba9616c6c9b757"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.770083 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 03:05:58.773329 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:58.774714 20643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:05:58.774858 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "08d6c8dd9349402f8fd9253bb9a5d240"
format_stamp: "Formatted at 2026-09-29 03:05:58 on dist-test-slave-530n"
server_key: "a8ad38c9134b454f78baa2c9c0491501"
server_key_iv: "38dd53c9aa3baa1d1fba9616c6c9b757"
server_key_version: "encryptionkey@0"
I20260929 03:05:58.775048 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:58.788151 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:58.789039 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:58.789314 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:58.790251 20643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:58.791756 20643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:58.791843 20643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:58.791949 20643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:58.792016 20643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:58.822299 20643 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:37553
I20260929 03:05:58.822350 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:37553 every 8 connection(s)
I20260929 03:05:58.823707 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:05:58.833736 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20643
I20260929 03:05:58.833920 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:05:58.834502 2870 external_mini_cluster.cc:1428] Setting key 828712e339616f65529088e3ea633f2b
I20260929 03:05:58.836478 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:58.826252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:33273 (local address 127.2.205.190:37761)
0929 03:05:58.826675 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:58.826681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:58.828029 (+ 1348us) server_negotiation.cc:407] Connection header received
0929 03:05:58.829917 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:58.829922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:58.830010 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:58.830116 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:58.831200 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.831899 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.833535 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.834154 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.834289 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:58.835636 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:58.835642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:58.835646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:05:58.835664 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:58.835708 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:05:58.835721 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:58.835792 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:58.836040 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:05:58.836133 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":199,"threads_started":1}
I20260929 03:05:58.837848 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--builtin_ntp_servers=127.2.205.148:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:05:58.838138 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:58.826921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:37761 (local address 127.2.205.129:33273)
0929 03:05:58.827852 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:58.827901 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:58.829626 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:58.830337 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:58.830346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:58.830442 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:58.831044 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:58.831060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.832005 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.832008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:58.833377 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:58.833391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:58.834365 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:58.834369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:58.834602 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:58.835327 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:05:58.835330 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:58.835375 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:05:58.835466 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:58.835814 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:58.835826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:58.835835 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:05:58.836038 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:05:58.836528 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":264,"threads_started":1}
I20260929 03:05:58.839895 20770 heartbeater.cc:347] Connected to a master server at 127.2.205.190:37761
I20260929 03:05:58.840267 20770 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:58.841260 20770 heartbeater.cc:511] Master 127.2.205.190:37761 requested a full tablet report, sending...
I20260929 03:05:58.843583 20584 ts_manager.cc:194] Registered new tserver with Master: 08d6c8dd9349402f8fd9253bb9a5d240 (127.2.205.129:37553)
I20260929 03:05:58.845212 20584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.129:33273
W20260929 03:05:59.161511 20774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:59.161796 20774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:59.161906 20774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:59.169090 20774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:59.169195 20774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:59.169337 20774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:05:59.178447 20774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:59.179831 20774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:59.181689 20774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:59.194657 20779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:59.194622 20780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:59.194576 20782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:59.195397 20774 server_base.cc:1034] running on GCE node
I20260929 03:05:59.195989 20774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:59.196938 20774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:59.198132 20774 hybrid_clock.cc:648] HybridClock initialized: now 1790651159198072 us; error 46 us; skew 500 ppm
I20260929 03:05:59.198438 20774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:05:59.200767 20774 webserver.cc:460] Webserver started at http://127.2.205.130:46139/ using document root <none> and password file <none>
I20260929 03:05:59.201450 20774 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:59.201551 20774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:59.201848 20774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:59.204643 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "4faf028628b54a9d8bcb68ff087b78b5"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "b04712497cea5285f3e4db901a3236ac"
server_key_iv: "adcd09d1c1fddf53080daf60aadf2efa"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.205430 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "4faf028628b54a9d8bcb68ff087b78b5"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "b04712497cea5285f3e4db901a3236ac"
server_key_iv: "adcd09d1c1fddf53080daf60aadf2efa"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.210547 20774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:05:59.214650 20788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.216382 20774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:05:59.216569 20774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "4faf028628b54a9d8bcb68ff087b78b5"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "b04712497cea5285f3e4db901a3236ac"
server_key_iv: "adcd09d1c1fddf53080daf60aadf2efa"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.216847 20774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:59.242578 20774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:59.243549 20774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:59.243873 20774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:59.244987 20774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:59.246767 20774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:59.246873 20774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.246997 20774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:59.247074 20774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.278398 20774 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:41683
I20260929 03:05:59.278488 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:41683 every 8 connection(s)
I20260929 03:05:59.279693 20774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:05:59.282435 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20774
I20260929 03:05:59.282629 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:05:59.283155 2870 external_mini_cluster.cc:1428] Setting key 9a6d386356c078afd9cef1ba30181c86
I20260929 03:05:59.286674 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--builtin_ntp_servers=127.2.205.148:36235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:05:59.295379 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.282338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:45073 (local address 127.2.205.190:37761)
0929 03:05:59.282568 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.282575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.284825 (+ 2250us) server_negotiation.cc:407] Connection header received
0929 03:05:59.287415 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.287420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.287511 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.287620 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:59.289136 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.289883 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.291957 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.292556 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.292706 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.294482 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:59.294488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:59.294491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:05:59.294515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:59.294553 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:05:59.294566 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:59.294649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.294943 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.295067 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:05:59.296639 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.282933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:37761 (local address 127.2.205.130:45073)
0929 03:05:59.284591 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.284646 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.287024 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.287964 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.287975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.288112 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:59.288897 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.288925 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.290072 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.290076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.291747 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.291764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.292848 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.292855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.293171 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.294024 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:05:59.294027 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:59.294093 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:05:59.294237 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:59.294671 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:59.294684 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:59.294693 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.294894 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.295393 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":729,"threads_started":1}
I20260929 03:05:59.298337 20901 heartbeater.cc:347] Connected to a master server at 127.2.205.190:37761
I20260929 03:05:59.298656 20901 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:59.299487 20901 heartbeater.cc:511] Master 127.2.205.190:37761 requested a full tablet report, sending...
I20260929 03:05:59.301148 20584 ts_manager.cc:194] Registered new tserver with Master: 4faf028628b54a9d8bcb68ff087b78b5 (127.2.205.130:41683)
I20260929 03:05:59.302196 20584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.130:45073
W20260929 03:05:59.611163 20904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:05:59.611470 20904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:05:59.611588 20904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:05:59.618832 20904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:05:59.618938 20904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:05:59.619047 20904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:05:59.628427 20904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:36235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:37761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:05:59.629762 20904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:05:59.631652 20904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:05:59.644567 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:59.644603 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:05:59.644670 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:05:59.645691 20904 server_base.cc:1034] running on GCE node
I20260929 03:05:59.646277 20904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:05:59.647219 20904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:05:59.648424 20904 hybrid_clock.cc:648] HybridClock initialized: now 1790651159648361 us; error 49 us; skew 500 ppm
I20260929 03:05:59.648717 20904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:05:59.650923 20904 webserver.cc:460] Webserver started at http://127.2.205.131:33219/ using document root <none> and password file <none>
I20260929 03:05:59.651594 20904 fs_manager.cc:359] Metadata directory not provided
I20260929 03:05:59.651676 20904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:05:59.651932 20904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:05:59.654644 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "391f6371e720457a914b659d0f2897bf"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "dfb2ac53c27402b3ad933137046931a8"
server_key_iv: "880af7916a239a48f72821a97d36f4ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.655354 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "391f6371e720457a914b659d0f2897bf"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "dfb2ac53c27402b3ad933137046931a8"
server_key_iv: "880af7916a239a48f72821a97d36f4ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.659962 20904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 03:05:59.663085 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.664359 20904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:05:59.664486 20904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "391f6371e720457a914b659d0f2897bf"
format_stamp: "Formatted at 2026-09-29 03:05:59 on dist-test-slave-530n"
server_key: "dfb2ac53c27402b3ad933137046931a8"
server_key_iv: "880af7916a239a48f72821a97d36f4ed"
server_key_version: "encryptionkey@0"
I20260929 03:05:59.664654 20904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:05:59.694589 20904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:05:59.695463 20904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:05:59.695734 20904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:05:59.696817 20904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:05:59.698359 20904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:05:59.698444 20904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.698534 20904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:05:59.698586 20904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:05:59.730295 20904 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:45245
I20260929 03:05:59.730370 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:45245 every 8 connection(s)
I20260929 03:05:59.731686 20904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:05:59.733211 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 20904
I20260929 03:05:59.733390 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:05:59.733963 2870 external_mini_cluster.cc:1428] Setting key f5988679e85e289987b91b1d2e431b82
I20260929 03:05:59.744282 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.734586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:45471 (local address 127.2.205.190:37761)
0929 03:05:59.734768 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.734773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.736518 (+ 1745us) server_negotiation.cc:407] Connection header received
0929 03:05:59.738374 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.738378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.738446 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.738527 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:59.739558 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.740316 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.741816 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.742224 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.742313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.743581 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:59.743587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:59.743590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:05:59.743609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:59.743646 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:05:59.743658 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:59.743734 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.743941 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.744034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:05:59.745380 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.735263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:37761 (local address 127.2.205.131:45471)
0929 03:05:59.736353 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.736408 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.738129 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.738715 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.738724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.738813 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:05:59.739412 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.739429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.740429 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.740432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.741692 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.741704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.742382 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.742386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.742583 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.743258 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:05:59.743261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:05:59.743300 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:05:59.743376 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:05:59.743751 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:05:59.743759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:05:59.743765 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.743916 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.744235 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260929 03:05:59.746724 21031 heartbeater.cc:347] Connected to a master server at 127.2.205.190:37761
I20260929 03:05:59.747008 21031 heartbeater.cc:464] Registering TS with master...
I20260929 03:05:59.747912 21031 heartbeater.cc:511] Master 127.2.205.190:37761 requested a full tablet report, sending...
I20260929 03:05:59.749325 20583 ts_manager.cc:194] Registered new tserver with Master: 391f6371e720457a914b659d0f2897bf (127.2.205.131:45245)
I20260929 03:05:59.750201 20583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.131:45471
I20260929 03:05:59.761514 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:05:59.770974 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.765250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.2.205.190:37761)
0929 03:05:59.765456 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.765463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.765843 (+ 380us) server_negotiation.cc:407] Connection header received
0929 03:05:59.765998 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.766003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.766087 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.766168 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:59.766669 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.767376 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.768805 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.769224 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.769331 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.770325 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:05:59.770331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:05:59.770334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:05:59.770350 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:05:59.770381 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:05:59.770393 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:05:59.770463 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.770598 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.770689 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:05:59.774313 20583 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:05:59.781008 20583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:05:59.804991 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.795308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38488)
0929 03:05:59.796487 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.796514 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.796643 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.797388 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.797395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.797427 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:59.797800 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.797814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.799906 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.799911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.802328 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.802343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.803993 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.804000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.804200 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.804255 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.804389 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.804580 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":240,"thread_start_us":853,"threads_started":1}
I20260929 03:05:59.805459 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.793658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:45245 (local address 127.0.0.1:44748)
0929 03:05:59.794374 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.794411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.794576 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.797768 (+ 3192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.797772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.797796 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:59.798272 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.798291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.800100 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.800105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.802825 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.802838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.804645 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.804652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.804832 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.804864 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.804924 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.805064 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":156,"thread_start_us":399,"threads_started":1}
I20260929 03:05:59.805497 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.793772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.2.205.130:41683)
0929 03:05:59.797329 (+ 3557us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.797337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.797368 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:05:59.797461 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.797468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.797712 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.798059 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:59.798577 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.799835 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.802757 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.804185 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.804325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.804564 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.804697 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.804888 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3373,"thread_start_us":3647,"threads_started":1}
I20260929 03:05:59.805748 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.793676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:41683 (local address 127.0.0.1:56116)
0929 03:05:59.796464 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.796482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.796584 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.797921 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.797925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.797949 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:59.798421 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.798431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.799999 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.800014 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.802547 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.802562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.804358 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.804365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.804515 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.804545 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.804654 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.804809 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"mutex_wait_us":114,"spinlock_wait_cycles":768,"thread_start_us":1526,"threads_started":1}
I20260929 03:05:59.805785 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.795691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.2.205.129:37553)
0929 03:05:59.796235 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.796244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.796624 (+ 380us) server_negotiation.cc:407] Connection header received
0929 03:05:59.796832 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.796841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.797141 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.798023 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:59.798257 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.799762 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.802535 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.803821 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.803940 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.804203 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.805288 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.805478 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260929 03:05:59.807238 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.794386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.2.205.131:45245)
0929 03:05:59.797206 (+ 2820us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.797214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.797244 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:05:59.797338 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.797346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.797588 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.797763 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:59.798482 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.799980 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.803045 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.804425 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.805437 (+ 1012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.805707 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.805779 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.806886 (+ 1107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2668,"thread_start_us":2515,"threads_started":1}
I20260929 03:05:59.809496 20836 tablet_service.cc:1467] Processing CreateTablet for tablet 6d104742bbb44bbdaa855c28031a738c (DEFAULT_TABLE table=test-table [id=5a3e5aef6eb94aabb7bd29a484421c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:59.809484 20705 tablet_service.cc:1467] Processing CreateTablet for tablet 6d104742bbb44bbdaa855c28031a738c (DEFAULT_TABLE table=test-table [id=5a3e5aef6eb94aabb7bd29a484421c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:59.810854 20836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d104742bbb44bbdaa855c28031a738c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:59.810916 20705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d104742bbb44bbdaa855c28031a738c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:59.812582 20966 tablet_service.cc:1467] Processing CreateTablet for tablet 6d104742bbb44bbdaa855c28031a738c (DEFAULT_TABLE table=test-table [id=5a3e5aef6eb94aabb7bd29a484421c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:05:59.814178 20966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d104742bbb44bbdaa855c28031a738c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:05:59.829574 21049 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Bootstrap starting.
I20260929 03:05:59.833547 21050 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Bootstrap starting.
I20260929 03:05:59.835144 21049 tablet_bootstrap.cc:654] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:59.835706 21051 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Bootstrap starting.
I20260929 03:05:59.836530 21049 log.cc:824] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:59.837250 21050 tablet_bootstrap.cc:654] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:59.838407 21050 log.cc:824] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:59.840055 21049 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: No bootstrap required, opened a new log
I20260929 03:05:59.840401 21049 ts_tablet_manager.cc:1397] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260929 03:05:59.841408 21051 tablet_bootstrap.cc:654] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Neither blocks nor log segments found. Creating new log.
I20260929 03:05:59.842664 21051 log.cc:824] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:05:59.843024 21050 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: No bootstrap required, opened a new log
I20260929 03:05:59.843304 21050 ts_tablet_manager.cc:1397] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 03:05:59.845861 21051 tablet_bootstrap.cc:492] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: No bootstrap required, opened a new log
I20260929 03:05:59.846117 21051 ts_tablet_manager.cc:1397] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 03:05:59.848430 20770 heartbeater.cc:503] Master 127.2.205.190:37761 was elected leader, sending a full tablet report...
I20260929 03:05:59.850383 21049 raft_consensus.cc:348] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.850811 21049 raft_consensus.cc:374] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:59.850896 21049 raft_consensus.cc:729] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faf028628b54a9d8bcb68ff087b78b5, State: Initialized, Role: FOLLOWER
I20260929 03:05:59.851084 21050 raft_consensus.cc:348] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.851498 21050 raft_consensus.cc:374] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:59.851593 21050 raft_consensus.cc:729] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d6c8dd9349402f8fd9253bb9a5d240, State: Initialized, Role: FOLLOWER
I20260929 03:05:59.851502 21049 consensus_queue.cc:260] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.852123 21050 consensus_queue.cc:260] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.856487 21050 ts_tablet_manager.cc:1428] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 03:05:59.856304 21051 raft_consensus.cc:348] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.856724 21051 raft_consensus.cc:374] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:05:59.856854 21051 raft_consensus.cc:729] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391f6371e720457a914b659d0f2897bf, State: Initialized, Role: FOLLOWER
I20260929 03:05:59.857563 20901 heartbeater.cc:503] Master 127.2.205.190:37761 was elected leader, sending a full tablet report...
I20260929 03:05:59.858173 21049 ts_tablet_manager.cc:1428] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
I20260929 03:05:59.858127 21051 consensus_queue.cc:260] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.859197 21031 heartbeater.cc:503] Master 127.2.205.190:37761 was elected leader, sending a full tablet report...
I20260929 03:05:59.859722 21051 ts_tablet_manager.cc:1428] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 03:05:59.882282 21055 raft_consensus.cc:482] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:05:59.882637 21055 raft_consensus.cc:504] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.884462 21055 leader_election.cc:288] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 391f6371e720457a914b659d0f2897bf (127.2.205.131:45245), 08d6c8dd9349402f8fd9253bb9a5d240 (127.2.205.129:37553)
I20260929 03:05:59.891304 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.884839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:58747 (local address 127.2.205.131:45245)
0929 03:05:59.885027 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.885032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.885578 (+ 546us) server_negotiation.cc:407] Connection header received
0929 03:05:59.885807 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.885814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.885902 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.886038 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:59.886653 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.887573 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.889628 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.890386 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.890543 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.890631 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.890934 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.891063 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:05:59.891376 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.885846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:59031 (local address 127.2.205.129:37553)
0929 03:05:59.886053 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.886059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.886210 (+ 151us) server_negotiation.cc:407] Connection header received
0929 03:05:59.886393 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.886398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.886475 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.886552 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:05:59.887123 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.887777 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.889682 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.890444 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.890589 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.890661 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.891007 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.891140 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:05:59.891412 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.885570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.2.205.130:59031)
0929 03:05:59.886125 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.886138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.886269 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.886582 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.886586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.886609 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:59.886893 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.886901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.887925 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.887930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.889479 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.889492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.890683 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.890692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.890833 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.890857 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.890931 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.891055 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":273,"threads_started":1}
I20260929 03:05:59.891466 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.884846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:45245 (local address 127.2.205.130:58747)
0929 03:05:59.885453 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 03:05:59.885469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:05:59.885628 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:05:59.886075 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:05:59.886080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:05:59.886106 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:05:59.886503 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.886513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.887707 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.887711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.889423 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:05:59.889439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.890582 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.890588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:05:59.890743 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.890767 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:05:59.890854 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:05:59.890984 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"spinlock_wait_cycles":18304,"thread_start_us":303,"threads_started":1}
I20260929 03:05:59.892524 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d104742bbb44bbdaa855c28031a738c" candidate_uuid: "4faf028628b54a9d8bcb68ff087b78b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" is_pre_election: true
I20260929 03:05:59.892617 20986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d104742bbb44bbdaa855c28031a738c" candidate_uuid: "4faf028628b54a9d8bcb68ff087b78b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391f6371e720457a914b659d0f2897bf" is_pre_election: true
I20260929 03:05:59.893211 20986 raft_consensus.cc:2454] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4faf028628b54a9d8bcb68ff087b78b5 in term 0.
I20260929 03:05:59.893211 20725 raft_consensus.cc:2454] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4faf028628b54a9d8bcb68ff087b78b5 in term 0.
I20260929 03:05:59.893893 20790 leader_election.cc:302] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d6c8dd9349402f8fd9253bb9a5d240, 4faf028628b54a9d8bcb68ff087b78b5; no voters:
I20260929 03:05:59.894258 21055 raft_consensus.cc:2781] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:05:59.894398 21055 raft_consensus.cc:482] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:05:59.894495 21055 raft_consensus.cc:3037] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:59.896682 21055 raft_consensus.cc:504] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.897279 21055 leader_election.cc:288] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [CANDIDATE]: Term 1 election: Requested vote from peers 391f6371e720457a914b659d0f2897bf (127.2.205.131:45245), 08d6c8dd9349402f8fd9253bb9a5d240 (127.2.205.129:37553)
I20260929 03:05:59.897781 20986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d104742bbb44bbdaa855c28031a738c" candidate_uuid: "4faf028628b54a9d8bcb68ff087b78b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391f6371e720457a914b659d0f2897bf"
I20260929 03:05:59.897831 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d104742bbb44bbdaa855c28031a738c" candidate_uuid: "4faf028628b54a9d8bcb68ff087b78b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d6c8dd9349402f8fd9253bb9a5d240"
I20260929 03:05:59.897970 20986 raft_consensus.cc:3037] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:59.898015 20725 raft_consensus.cc:3037] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:05:59.900166 20986 raft_consensus.cc:2454] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4faf028628b54a9d8bcb68ff087b78b5 in term 1.
I20260929 03:05:59.900187 20725 raft_consensus.cc:2454] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4faf028628b54a9d8bcb68ff087b78b5 in term 1.
I20260929 03:05:59.900556 20790 leader_election.cc:302] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391f6371e720457a914b659d0f2897bf, 4faf028628b54a9d8bcb68ff087b78b5; no voters:
I20260929 03:05:59.900875 21055 raft_consensus.cc:2781] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:05:59.901312 21055 raft_consensus.cc:686] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4faf028628b54a9d8bcb68ff087b78b5, State: Running, Role: LEADER
I20260929 03:05:59.901831 21055 consensus_queue.cc:237] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } }
I20260929 03:05:59.906491 20584 catalog_manager.cc:5495] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4faf028628b54a9d8bcb68ff087b78b5 (127.2.205.130). New cstate: current_term: 1 leader_uuid: "4faf028628b54a9d8bcb68ff087b78b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4faf028628b54a9d8bcb68ff087b78b5" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 41683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 } health_report { overall_health: UNKNOWN } } }
I20260929 03:05:59.939065 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.933645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.2.205.130:41683)
0929 03:05:59.933838 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.933842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.933945 (+ 103us) server_negotiation.cc:407] Connection header received
0929 03:05:59.934131 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.934135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.934214 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.934346 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:05:59.934714 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.935464 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.936912 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.937337 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.937434 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:05:59.937988 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:05:59.938528 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:05:59.938654 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:05:59.938766 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:05:59.938836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:05:59.947532 20986 raft_consensus.cc:1261] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 1 FOLLOWER]: Refusing update from remote peer 4faf028628b54a9d8bcb68ff087b78b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:59.947531 20725 raft_consensus.cc:1261] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 1 FOLLOWER]: Refusing update from remote peer 4faf028628b54a9d8bcb68ff087b78b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:05:59.948452 21060 consensus_queue.cc:1035] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391f6371e720457a914b659d0f2897bf" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:05:59.948801 21055 consensus_queue.cc:1035] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d6c8dd9349402f8fd9253bb9a5d240" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:05:59.958966 21063 mvcc.cc:205] Tried to move back new op lower bound from 7334507151134605312 to 7334507150962409472. Current Snapshot: MvccSnapshot[applied={T|T < 7334507151134605312}]
I20260929 03:05:59.972642 20583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:59.972945 20583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:05:59.976421 20583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5fa4e324e514a4e8df0cae46265af02: Sending DeleteTablet for 3 replicas of tablet 6d104742bbb44bbdaa855c28031a738c
I20260929 03:05:59.977475 20836 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d104742bbb44bbdaa855c28031a738c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:59 UTC) from {username='slave'} at 127.0.0.1:56116
I20260929 03:05:59.977578 20705 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d104742bbb44bbdaa855c28031a738c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:59 UTC) from {username='slave'} at 127.0.0.1:38488
I20260929 03:05:59.978477 20966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d104742bbb44bbdaa855c28031a738c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:05:59 UTC) from {username='slave'} at 127.0.0.1:44748
I20260929 03:05:59.978615 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20643
I20260929 03:05:59.996217 21074 tablet_replica.cc:330] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: stopping tablet replica
I20260929 03:05:59.996511 21073 tablet_replica.cc:330] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: stopping tablet replica
I20260929 03:05:59.997025 21073 raft_consensus.cc:2229] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:05:59.997805 21073 raft_consensus.cc:2258] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:59.998214 21074 raft_consensus.cc:2229] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:05:59.998772 21075 tablet_replica.cc:330] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: stopping tablet replica
I20260929 03:05:59.998796 21074 raft_consensus.cc:2258] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:05:59.999244 21075 raft_consensus.cc:2229] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:00.000229 21075 raft_consensus.cc:2258] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:00.000554 21073 ts_tablet_manager.cc:1905] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:00.001823 21074 ts_tablet_manager.cc:1905] T 6d104742bbb44bbdaa855c28031a738c P 08d6c8dd9349402f8fd9253bb9a5d240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:00.001957 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:05:59.980014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.2.205.129:37553)
0929 03:05:59.980195 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:05:59.980202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:05:59.990553 (+ 10351us) server_negotiation.cc:407] Connection header received
0929 03:05:59.991119 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:05:59.991127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:05:59.991262 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:05:59.991344 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:05:59.992163 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.993542 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.996863 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:05:59.997471 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:05:59.997570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:00.000141 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:00.000152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:00.000169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:00.001027 (+ 858us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:00.001089 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:00.001114 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:00.001206 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:00.001446 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:06:00.001579 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:06:00.003096 20765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:00.004109 21075 ts_tablet_manager.cc:1905] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:00.008936 21073 ts_tablet_manager.cc:1918] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:00.009136 21073 log.cc:1196] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/6d104742bbb44bbdaa855c28031a738c
I20260929 03:06:00.009673 21073 ts_tablet_manager.cc:1939] T 6d104742bbb44bbdaa855c28031a738c P 4faf028628b54a9d8bcb68ff087b78b5: Deleting consensus metadata
I20260929 03:06:00.011266 21075 ts_tablet_manager.cc:1918] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:00.011461 21075 log.cc:1196] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/6d104742bbb44bbdaa855c28031a738c
I20260929 03:06:00.011986 21075 ts_tablet_manager.cc:1939] T 6d104742bbb44bbdaa855c28031a738c P 391f6371e720457a914b659d0f2897bf: Deleting consensus metadata
I20260929 03:06:00.012264 20570 catalog_manager.cc:4841] TS 4faf028628b54a9d8bcb68ff087b78b5 (127.2.205.130:41683): tablet 6d104742bbb44bbdaa855c28031a738c (table test-table [id=5a3e5aef6eb94aabb7bd29a484421c73]) successfully deleted
I20260929 03:06:00.014077 20569 catalog_manager.cc:4841] TS 391f6371e720457a914b659d0f2897bf (127.2.205.131:45245): tablet 6d104742bbb44bbdaa855c28031a738c (table test-table [id=5a3e5aef6eb94aabb7bd29a484421c73]) successfully deleted
I20260929 03:06:00.100951 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20643
I20260929 03:06:00.115957 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20774
W20260929 03:06:00.116112 20569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:37553 (error 108)
I20260929 03:06:00.118330 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.117871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38512)
0929 03:06:00.118055 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.118180 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 03:06:00.118618 20569 catalog_manager.cc:4584] TS 08d6c8dd9349402f8fd9253bb9a5d240 (127.2.205.129:37553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d104742bbb44bbdaa855c28031a738c: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
I20260929 03:06:00.124330 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.116877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.2.205.130:41683)
0929 03:06:00.117066 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:00.117072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:00.117095 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:06:00.117164 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:00.117169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:00.117295 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:00.117413 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:00.117895 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:00.118990 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:00.120300 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:00.121051 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:00.121211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:00.122433 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:00.122445 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:00.122467 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:00.123368 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:00.123447 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:00.123477 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:00.123626 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:00.123802 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 03:06:00.123926 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:06:00.125173 20896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:00.172408 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.171821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38518)
0929 03:06:00.172095 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.172220 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 03:06:00.173596 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.173185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38524)
0929 03:06:00.173369 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.173471 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:06:00.234447 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.233829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38540)
0929 03:06:00.234104 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.234209 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 03:06:00.235497 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.235134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38544)
0929 03:06:00.235292 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.235372 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:06:00.238730 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20774
I20260929 03:06:00.254889 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20904
I20260929 03:06:00.262203 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.255683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.2.205.131:45245)
0929 03:06:00.255872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:00.255879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:00.255942 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:06:00.256101 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:00.256107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:00.256212 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:00.256321 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:00.256873 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:00.257693 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:00.259116 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:00.259569 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:00.259679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:00.260888 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:00.260898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:00.260913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:00.261525 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:00.261574 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:00.261594 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:00.261684 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:00.261785 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 03:06:00.261875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:06:00.262938 21026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:00.316332 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.315687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38560)
0929 03:06:00.316012 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.316138 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 03:06:00.317644 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:00.317158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:37553 (local address 127.0.0.1:38566)
0929 03:06:00.317385 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:00.317484 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 03:06:00.351614 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20904
I20260929 03:06:00.364738 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 20553
I20260929 03:06:00.365944 20614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:00.474467 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 20553
2026-09-29T03:06:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2719 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:00 dist-test-slave-530n krb5kdc[21084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:00 dist-test-slave-530n krb5kdc[21084](info): set up 2 sockets
Sep 29 03:06:00 dist-test-slave-530n krb5kdc[21084](info): commencing operation
krb5kdc: starting...
W20260929 03:06:01.497004 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:01 dist-test-slave-530n krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:01.735383 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45023
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:02.069674 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:02.069962 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:02.070048 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:02.077080 21097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:02.077216 21097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:02.077289 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:02.077349 21097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:02.077394 21097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:02.087287 21097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:02.088706 21097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:02.090116 21097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:02.102600 21102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:02.102666 21103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:02.102579 21105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:02.103619 21097 server_base.cc:1034] running on GCE node
I20260929 03:06:02.104518 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:02.106266 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:02.107491 21097 hybrid_clock.cc:648] HybridClock initialized: now 1790651162107436 us; error 47 us; skew 500 ppm
I20260929 03:06:02.107896 21097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:02 dist-test-slave-530n krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:02.111150 21097 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:02.112815 21097 webserver.cc:460] Webserver started at http://127.2.205.190:42535/ using document root <none> and password file <none>
I20260929 03:06:02.113543 21097 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:02.113656 21097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:02.113978 21097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:02.117233 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "96288723d83a44f4872ef233ea45581b"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "c947136faf5c29cf21e46459c2f458b9"
server_key_iv: "c4f8c7c548c71b4508571c7586ce065e"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.118031 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "96288723d83a44f4872ef233ea45581b"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "c947136faf5c29cf21e46459c2f458b9"
server_key_iv: "c4f8c7c548c71b4508571c7586ce065e"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.123862 21097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:06:02.127914 21112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:02.129575 21097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:02.129758 21097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "96288723d83a44f4872ef233ea45581b"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "c947136faf5c29cf21e46459c2f458b9"
server_key_iv: "c4f8c7c548c71b4508571c7586ce065e"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.129973 21097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:02.152917 21097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:02.157191 21097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:02.157462 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:02.174917 21097 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:45023
I20260929 03:06:02.175006 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:45023 every 8 connection(s)
I20260929 03:06:02.176235 21097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:02.179373 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21097
I20260929 03:06:02.179553 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:02.179986 2870 external_mini_cluster.cc:1428] Setting key e36d3945857603e50bce4e73e8de7293
I20260929 03:06:02.180197 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:02.195328 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: Bootstrap starting.
Sep 29 03:06:02 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:02.199231 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:02.200646 21164 log.cc:824] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:02.204463 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: No bootstrap required, opened a new log
I20260929 03:06:02.206463 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:02.181814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.2.205.190:45023)
0929 03:06:02.182620 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:02.182633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:02.182676 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:06:02.184970 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:02.184998 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:02.185458 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:02.185858 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:02.187844 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.189289 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.191234 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.192051 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.192221 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:02.197187 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:02.197200 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:02.197222 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:02.197278 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:02.201242 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:02.202341 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:02.202350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:02.202360 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:02.202572 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:02.202936 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:02.202940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:02.202943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:02.203722 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:02.204032 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:02.204466 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 03:06:02.205004 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":375,"threads_started":1}
I20260929 03:06:02.212986 21164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } }
I20260929 03:06:02.213449 21164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:02.213578 21164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96288723d83a44f4872ef233ea45581b, State: Initialized, Role: FOLLOWER
I20260929 03:06:02.214329 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } }
I20260929 03:06:02.214594 21164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:02.214708 21164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:02.214847 21164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:02.217226 21164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } }
I20260929 03:06:02.217800 21164 leader_election.cc:302] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96288723d83a44f4872ef233ea45581b; no voters:
I20260929 03:06:02.218421 21164 leader_election.cc:288] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:02.218613 21169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:02.219319 21169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [term 1 LEADER]: Becoming Leader. State: Replica: 96288723d83a44f4872ef233ea45581b, State: Running, Role: LEADER
I20260929 03:06:02.219961 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } }
I20260929 03:06:02.220604 21164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:02.223199 21171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96288723d83a44f4872ef233ea45581b. Latest consensus state: current_term: 1 leader_uuid: "96288723d83a44f4872ef233ea45581b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } } }
I20260929 03:06:02.223861 21171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:02.223078 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96288723d83a44f4872ef233ea45581b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96288723d83a44f4872ef233ea45581b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45023 } } }
I20260929 03:06:02.224217 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:02.229847 21176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:02.235787 21176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:02.245887 21176 catalog_manager.cc:1347] Generated new cluster ID: 48915a09eccd44668256c2beedd1e484
I20260929 03:06:02.246038 21176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:02.258380 21176 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:02.259716 21176 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:02.268982 21176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: Generated new TSK 0
I20260929 03:06:02.269860 21176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:06:02.342574 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:02.682315 21190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:02.682602 21190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:02.682718 21190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:02.690781 21190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:02.690891 21190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:02.691057 21190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:06:02.700901 21190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:02.702212 21190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:02.703549 21190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:02.716777 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:02.716827 21195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:02.716828 21196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:02.717638 21190 server_base.cc:1034] running on GCE node
I20260929 03:06:02.718258 21190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:02.719295 21190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:02.720541 21190 hybrid_clock.cc:648] HybridClock initialized: now 1790651162720475 us; error 60 us; skew 500 ppm
I20260929 03:06:02.720945 21190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:02 dist-test-slave-530n krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:02.724588 21190 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:06:02.726467 21190 webserver.cc:460] Webserver started at http://127.2.205.129:37887/ using document root <none> and password file <none>
I20260929 03:06:02.727308 21190 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:02.727439 21190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:02.727780 21190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:02.731002 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "31ddfa83759f4646be74be623c691bf7"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "96fb8ff6ec89c79cacbcafced9dcbc75"
server_key_iv: "1f138c046e252264e1f87bf031e188c9"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.731786 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "31ddfa83759f4646be74be623c691bf7"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "96fb8ff6ec89c79cacbcafced9dcbc75"
server_key_iv: "1f138c046e252264e1f87bf031e188c9"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.737155 21190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 03:06:02.740685 21205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:02.742077 21190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:02.742220 21190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "31ddfa83759f4646be74be623c691bf7"
format_stamp: "Formatted at 2026-09-29 03:06:02 on dist-test-slave-530n"
server_key: "96fb8ff6ec89c79cacbcafced9dcbc75"
server_key_iv: "1f138c046e252264e1f87bf031e188c9"
server_key_version: "encryptionkey@0"
I20260929 03:06:02.742409 21190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:02.772460 21190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:02.777273 21190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:02.777571 21190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:02.778600 21190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:02.780020 21190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:02.780102 21190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:02.780241 21190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:02.780294 21190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:02.812084 21190 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:40535
I20260929 03:06:02.812144 21317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:40535 every 8 connection(s)
I20260929 03:06:02.813453 21190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:06:02.820147 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21190
I20260929 03:06:02.820343 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:06:02.820876 2870 external_mini_cluster.cc:1428] Setting key bcd1a5dcc6a3edb6869685e4f3f6965f
Sep 29 03:06:02 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:02.836546 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:02.816199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:47873 (local address 127.2.205.190:45023)
0929 03:06:02.816700 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:02.816705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:02.817836 (+ 1131us) server_negotiation.cc:407] Connection header received
0929 03:06:02.819842 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:02.819849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:02.819958 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:02.820098 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:02.822156 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.823135 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.824872 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.825452 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.825586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:02.829778 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:02.829785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:02.829790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:02.829833 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:02.833109 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:02.834015 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:02.834021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:02.834024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:02.834169 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:02.834641 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:02.834646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:02.834648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:02.835027 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:02.835164 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:02.835955 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 03:06:02.836193 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":250,"threads_started":1}
I20260929 03:06:02.837493 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:02.816886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45023 (local address 127.2.205.129:47873)
0929 03:06:02.817647 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:02.817691 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:02.819482 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:02.820365 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:02.820380 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:02.821137 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:02.821944 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:02.821972 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.823335 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.823339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:02.824694 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:02.824708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:02.825737 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:02.825742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:02.825973 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:02.826684 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:02.826709 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:02.829445 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:02.833303 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:02.833313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:02.833329 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:02.833848 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:02.834306 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:02.834310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:02.834312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:02.834514 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:02.835178 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:02.835185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:02.835553 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 03:06:02.835796 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 03:06:02.836255 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":290,"threads_started":1}
I20260929 03:06:02.839172 21318 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45023
I20260929 03:06:02.839469 21318 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:02.840224 21318 heartbeater.cc:511] Master 127.2.205.190:45023 requested a full tablet report, sending...
I20260929 03:06:02.842447 21129 ts_manager.cc:194] Registered new tserver with Master: 31ddfa83759f4646be74be623c691bf7 (127.2.205.129:40535)
I20260929 03:06:02.843912 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:47873
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:06:02.875559 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:03.204850 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:03.205206 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:03.205389 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:03.213100 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:03.213248 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:03.213407 21324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:06:03.223701 21324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:03.225126 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:03.226504 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:03.241742 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:03.241711 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:03.241936 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:03.242415 21324 server_base.cc:1034] running on GCE node
I20260929 03:06:03.243116 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:03.244081 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:03.245276 21324 hybrid_clock.cc:648] HybridClock initialized: now 1790651163245223 us; error 42 us; skew 500 ppm
I20260929 03:06:03.245597 21324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:03 dist-test-slave-530n krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:03.248649 21324 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:06:03.250034 21324 webserver.cc:460] Webserver started at http://127.2.205.130:40597/ using document root <none> and password file <none>
I20260929 03:06:03.250687 21324 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:03.250787 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:03.251044 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:03.253923 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "bebe105067264ce0a9fe9870a3c85815"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "086b20ab26057ebf4bcc33cf582e48a4"
server_key_iv: "37468cee8ec05aef3896d28fd505b5d9"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.254650 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "bebe105067264ce0a9fe9870a3c85815"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "086b20ab26057ebf4bcc33cf582e48a4"
server_key_iv: "37468cee8ec05aef3896d28fd505b5d9"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.260066 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 03:06:03.263468 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.264840 21324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:06:03.264971 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "bebe105067264ce0a9fe9870a3c85815"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "086b20ab26057ebf4bcc33cf582e48a4"
server_key_iv: "37468cee8ec05aef3896d28fd505b5d9"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.265141 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:03.301491 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:03.305984 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:03.306259 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:03.307312 21324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:03.308863 21324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:03.308940 21324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.309044 21324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:03.309088 21324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.340545 21324 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:33695
I20260929 03:06:03.340616 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:33695 every 8 connection(s)
I20260929 03:06:03.341991 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:06:03.350359 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21324
I20260929 03:06:03.350525 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:06:03.350960 2870 external_mini_cluster.cc:1428] Setting key 22410a810c2f549561e619e57204628e
Sep 29 03:06:03 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:03.362727 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.344877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:58113 (local address 127.2.205.190:45023)
0929 03:06:03.345354 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.345358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.346431 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 03:06:03.348404 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.348410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.348495 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.348604 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:03.350218 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.351010 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.352708 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.353216 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.353316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.357191 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:03.357197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:03.357200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:03.357233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:03.359661 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.360390 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.360394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.360396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.360531 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.361007 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.361010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.361012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.361285 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:03.361416 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.362119 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.362348 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
I20260929 03:06:03.363749 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.345456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45023 (local address 127.2.205.130:58113)
0929 03:06:03.346249 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:03.346290 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:03.348094 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:03.348921 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:03.348930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:03.349464 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:03.350029 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.350046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.351170 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.351176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.352564 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.352579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.353452 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.353461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.353757 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.354461 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:03.354495 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:03.356957 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:03.359805 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:03.359814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:03.359832 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:03.360258 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:03.360645 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:03.360650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:03.360654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:03.360889 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:03.361414 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:03.361422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:03.361761 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 03:06:03.361996 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 03:06:03.362541 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":327,"threads_started":1}
I20260929 03:06:03.365296 21452 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45023
I20260929 03:06:03.365599 21452 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:03.366299 21452 heartbeater.cc:511] Master 127.2.205.190:45023 requested a full tablet report, sending...
I20260929 03:06:03.367810 21129 ts_manager.cc:194] Registered new tserver with Master: bebe105067264ce0a9fe9870a3c85815 (127.2.205.130:33695)
I20260929 03:06:03.368788 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:58113
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:06:03.404017 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:03.734980 21458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:03.735287 21458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:03.735383 21458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:03.742671 21458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:03.742794 21458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:03.742908 21458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:06:03.752821 21458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:45023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:03.754165 21458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:03.755471 21458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:03.767840 21463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:03.767882 21464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:03.767903 21466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:03.768707 21458 server_base.cc:1034] running on GCE node
I20260929 03:06:03.769345 21458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:03.770280 21458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:03.771497 21458 hybrid_clock.cc:648] HybridClock initialized: now 1790651163771449 us; error 34 us; skew 500 ppm
I20260929 03:06:03.771793 21458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:03 dist-test-slave-530n krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:03.777794 21458 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:06:03.779302 21458 webserver.cc:460] Webserver started at http://127.2.205.131:44295/ using document root <none> and password file <none>
I20260929 03:06:03.779978 21458 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:03.780087 21458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:03.780365 21458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:03.783452 21458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "28079dd3892345c1afab5fc32aeacf2c"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "69d281664e9bda9304584e03b32e64f6"
server_key_iv: "40dd67ffc87c754a79c81ddd03dd7f07"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.784255 21458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "28079dd3892345c1afab5fc32aeacf2c"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "69d281664e9bda9304584e03b32e64f6"
server_key_iv: "40dd67ffc87c754a79c81ddd03dd7f07"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.789831 21458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 03:06:03.793699 21473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.795324 21458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:03.795521 21458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "28079dd3892345c1afab5fc32aeacf2c"
format_stamp: "Formatted at 2026-09-29 03:06:03 on dist-test-slave-530n"
server_key: "69d281664e9bda9304584e03b32e64f6"
server_key_iv: "40dd67ffc87c754a79c81ddd03dd7f07"
server_key_version: "encryptionkey@0"
I20260929 03:06:03.795729 21458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:03.822095 21458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:03.826638 21458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:03.826923 21458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:03.828001 21458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:03.829494 21458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:03.829568 21458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.829696 21458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:03.829743 21458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:03.847180 21318 heartbeater.cc:503] Master 127.2.205.190:45023 was elected leader, sending a full tablet report...
I20260929 03:06:03.860278 21458 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:46069
I20260929 03:06:03.860321 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:46069 every 8 connection(s)
I20260929 03:06:03.861568 21458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:06:03.869177 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21458
I20260929 03:06:03.869372 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:06:03.869885 2870 external_mini_cluster.cc:1428] Setting key 43f8ab4c64b1f0b92e72642999044edc
Sep 29 03:06:03 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:03.881563 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.863974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:39479 (local address 127.2.205.190:45023)
0929 03:06:03.864686 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.864692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.865451 (+ 759us) server_negotiation.cc:407] Connection header received
0929 03:06:03.867222 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.867226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.867292 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.867388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:03.868832 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.869957 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.871940 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.872331 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.872445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.875779 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:03.875784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:03.875787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:03.875819 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:03.878396 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.879146 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.879150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.879152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.879281 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.879718 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.879722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.879724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.880013 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:03.880157 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.880928 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.881187 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":458,"threads_started":1}
I20260929 03:06:03.882540 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.864546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:45023 (local address 127.2.205.131:39479)
0929 03:06:03.865279 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:03.865321 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:03.866940 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:03.867576 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:03.867597 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:03.868085 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:03.868640 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.868658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.870100 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.870105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.871813 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.871836 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.872551 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.872558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.872856 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.873433 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:03.873454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:03.875568 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:03.878562 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:03.878572 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:03.878589 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:03.879017 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:03.879397 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:03.879402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:03.879406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:03.879600 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:03.880149 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:03.880159 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:03.880515 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 03:06:03.880794 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 03:06:03.881350 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":300,"threads_started":1}
I20260929 03:06:03.884202 21586 heartbeater.cc:347] Connected to a master server at 127.2.205.190:45023
I20260929 03:06:03.884546 21586 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:03.885462 21586 heartbeater.cc:511] Master 127.2.205.190:45023 requested a full tablet report, sending...
I20260929 03:06:03.886951 21128 ts_manager.cc:194] Registered new tserver with Master: 28079dd3892345c1afab5fc32aeacf2c (127.2.205.131:46069)
I20260929 03:06:03.887859 21128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:39479
I20260929 03:06:03.898178 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:06:03.916365 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.902431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.2.205.190:45023)
0929 03:06:03.902607 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.902612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.903037 (+ 425us) server_negotiation.cc:407] Connection header received
0929 03:06:03.903181 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.903186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.903256 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.903348 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:03.904834 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.905594 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.907014 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.907434 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.907538 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.910555 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:03.910561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:03.910564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:03.910624 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:03.913694 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.914574 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.914589 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.914592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.914720 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:03.915122 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:03.915125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:03.915127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:03.915444 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:03.915660 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.915970 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.916129 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:06:03.920898 21128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:06:03.928294 21128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:06:03.977731 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.943260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.2.205.131:46069)
0929 03:06:03.969408 (+ 26148us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.969418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.969455 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:06:03.969557 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.969576 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.969831 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.970968 (+ 1137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:03.971228 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.972656 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.975208 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.976336 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.976458 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.976694 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.977192 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.977366 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25935,"thread_start_us":25924,"threads_started":1}
I20260929 03:06:03.977810 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.942528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:46069 (local address 127.0.0.1:40796)
0929 03:06:03.968372 (+ 25844us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:03.968390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:03.968528 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:03.970069 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:03.970077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:03.970124 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:03.970772 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.970787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.972818 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.972823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.975068 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.975080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.976841 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.976847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.977005 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.977034 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:06:03.977127 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:06:03.977315 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25709,"mutex_wait_us":27,"spinlock_wait_cycles":6912,"thread_start_us":25566,"threads_started":1}
I20260929 03:06:03.978266 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.942099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:33695 (local address 127.0.0.1:45508)
0929 03:06:03.967286 (+ 25187us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:03.967343 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:03.967604 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:03.970077 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:03.970083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:03.970129 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:03.970779 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.970794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.972445 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.972449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.974965 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.974980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.976522 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.976528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.976707 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.976785 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 03:06:03.977312 (+ 527us) client_negotiation.cc:240] Negotiation successful
0929 03:06:03.977419 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24875,"mutex_wait_us":23,"spinlock_wait_cycles":9472,"thread_start_us":27520,"threads_started":1}
I20260929 03:06:03.978479 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.943201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.2.205.129:40535)
0929 03:06:03.966378 (+ 23177us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.966390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.970048 (+ 3658us) server_negotiation.cc:407] Connection header received
0929 03:06:03.970444 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.970452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.970681 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.970846 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:03.971682 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.973157 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.975888 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.977149 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.977278 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.977480 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.978022 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.978181 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22946,"thread_start_us":23203,"threads_started":1}
I20260929 03:06:03.979027 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.942413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.2.205.130:33695)
0929 03:06:03.969257 (+ 26844us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:03.969265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:03.969296 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:06:03.969397 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:03.969405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:03.969849 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:03.970954 (+ 1105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:03.971173 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.972305 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.975153 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.976326 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.976452 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.976669 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:03.978515 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0929 03:06:03.978700 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26690,"thread_start_us":26570,"threads_started":1}
I20260929 03:06:03.979167 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:03.942637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:40535 (local address 127.0.0.1:53530)
0929 03:06:03.969898 (+ 27261us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:03.969915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:03.970163 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:03.971126 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:03.971130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:03.971153 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:03.971492 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.971503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.973332 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.973337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.975707 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:03.975721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:03.977633 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:03.977639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:03.977812 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:03.977842 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 03:06:03.978657 (+ 815us) client_negotiation.cc:240] Negotiation successful
0929 03:06:03.978786 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27123,"mutex_wait_us":111,"thread_start_us":25967,"threads_started":1}
I20260929 03:06:03.981971 21521 tablet_service.cc:1467] Processing CreateTablet for tablet daa13f375a49431fa1c4e4ea5f22097d (DEFAULT_TABLE table=test-table [id=b493188833074d30939a7e577fd91361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:03.982553 21253 tablet_service.cc:1467] Processing CreateTablet for tablet daa13f375a49431fa1c4e4ea5f22097d (DEFAULT_TABLE table=test-table [id=b493188833074d30939a7e577fd91361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:03.983765 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa13f375a49431fa1c4e4ea5f22097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:03.983865 21253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa13f375a49431fa1c4e4ea5f22097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:03.985565 21387 tablet_service.cc:1467] Processing CreateTablet for tablet daa13f375a49431fa1c4e4ea5f22097d (DEFAULT_TABLE table=test-table [id=b493188833074d30939a7e577fd91361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:03.986999 21387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa13f375a49431fa1c4e4ea5f22097d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:03.995851 21605 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Bootstrap starting.
I20260929 03:06:03.998121 21606 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Bootstrap starting.
I20260929 03:06:03.999719 21605 tablet_bootstrap.cc:654] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:04.001269 21607 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Bootstrap starting.
I20260929 03:06:04.001633 21606 tablet_bootstrap.cc:654] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:04.002198 21605 log.cc:824] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:04.003016 21606 log.cc:824] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:04.004676 21607 tablet_bootstrap.cc:654] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:04.005434 21605 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: No bootstrap required, opened a new log
I20260929 03:06:04.005693 21607 log.cc:824] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:04.005749 21605 ts_tablet_manager.cc:1397] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 03:06:04.006156 21606 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: No bootstrap required, opened a new log
I20260929 03:06:04.006474 21606 ts_tablet_manager.cc:1397] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 03:06:04.008126 21607 tablet_bootstrap.cc:492] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: No bootstrap required, opened a new log
I20260929 03:06:04.008364 21607 ts_tablet_manager.cc:1397] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 03:06:04.015969 21607 raft_consensus.cc:348] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.016376 21607 raft_consensus.cc:374] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:04.016494 21607 raft_consensus.cc:729] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebe105067264ce0a9fe9870a3c85815, State: Initialized, Role: FOLLOWER
I20260929 03:06:04.016673 21605 raft_consensus.cc:348] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.016780 21606 raft_consensus.cc:348] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.017184 21606 raft_consensus.cc:374] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:04.017184 21605 raft_consensus.cc:374] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:04.017280 21605 raft_consensus.cc:729] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ddfa83759f4646be74be623c691bf7, State: Initialized, Role: FOLLOWER
I20260929 03:06:04.017323 21606 raft_consensus.cc:729] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28079dd3892345c1afab5fc32aeacf2c, State: Initialized, Role: FOLLOWER
I20260929 03:06:04.017168 21607 consensus_queue.cc:260] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.017927 21606 consensus_queue.cc:260] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.017931 21605 consensus_queue.cc:260] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.019178 21586 heartbeater.cc:503] Master 127.2.205.190:45023 was elected leader, sending a full tablet report...
I20260929 03:06:04.019996 21606 ts_tablet_manager.cc:1428] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260929 03:06:04.022037 21607 ts_tablet_manager.cc:1428] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260929 03:06:04.022608 21452 heartbeater.cc:503] Master 127.2.205.190:45023 was elected leader, sending a full tablet report...
I20260929 03:06:04.024870 21605 ts_tablet_manager.cc:1428] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
W20260929 03:06:04.066089 21319 tablet.cc:2369] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:06:04.094256 21453 tablet.cc:2369] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:06:04.113302 21587 tablet.cc:2369] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:04.128844 21611 raft_consensus.cc:482] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:06:04.129248 21611 raft_consensus.cc:504] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.131273 21611 leader_election.cc:288] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28079dd3892345c1afab5fc32aeacf2c (127.2.205.131:46069), 31ddfa83759f4646be74be623c691bf7 (127.2.205.129:40535)
I20260929 03:06:04.138228 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.131760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:35053 (local address 127.2.205.131:46069)
0929 03:06:04.131993 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.132001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.132675 (+ 674us) server_negotiation.cc:407] Connection header received
0929 03:06:04.132849 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.132853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.132921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.133000 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:04.133754 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.134569 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.136403 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.137235 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.137389 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.137468 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.137844 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.137971 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 03:06:04.138269 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.131878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:54579 (local address 127.2.205.129:40535)
0929 03:06:04.132056 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.132062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.132563 (+ 501us) server_negotiation.cc:407] Connection header received
0929 03:06:04.132793 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.132799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.132869 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.132951 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:04.133698 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.134480 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.136414 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.137260 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.137406 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.137481 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.137885 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.138017 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:06:04.138379 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:46069 (local address 127.2.205.130:35053)
0929 03:06:04.132541 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:04.132556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:04.132696 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:04.133045 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:04.133049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:04.133078 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:04.133542 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:04.133552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.134717 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.134722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:04.136252 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:04.136264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.137508 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.137516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:04.137683 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.137709 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:06:04.137790 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:06:04.137936 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":7,"thread_start_us":595,"threads_started":1}
I20260929 03:06:04.138772 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.131574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:40535 (local address 127.2.205.130:54579)
0929 03:06:04.132434 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:04.132450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:04.132610 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:04.132990 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:04.132995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:04.133022 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:04.133525 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:04.133539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.134612 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.134618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:04.136243 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:04.136256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.137546 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.137553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:04.137721 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.137748 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:06:04.137826 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:06:04.137965 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":84,"thread_start_us":529,"threads_started":1}
I20260929 03:06:04.139496 21541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa13f375a49431fa1c4e4ea5f22097d" candidate_uuid: "bebe105067264ce0a9fe9870a3c85815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28079dd3892345c1afab5fc32aeacf2c" is_pre_election: true
I20260929 03:06:04.139755 21273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa13f375a49431fa1c4e4ea5f22097d" candidate_uuid: "bebe105067264ce0a9fe9870a3c85815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ddfa83759f4646be74be623c691bf7" is_pre_election: true
I20260929 03:06:04.140130 21541 raft_consensus.cc:2454] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebe105067264ce0a9fe9870a3c85815 in term 0.
I20260929 03:06:04.140276 21273 raft_consensus.cc:2454] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebe105067264ce0a9fe9870a3c85815 in term 0.
I20260929 03:06:04.140955 21340 leader_election.cc:302] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28079dd3892345c1afab5fc32aeacf2c, bebe105067264ce0a9fe9870a3c85815; no voters:
I20260929 03:06:04.141398 21611 raft_consensus.cc:2781] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:06:04.141534 21611 raft_consensus.cc:482] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:06:04.141654 21611 raft_consensus.cc:3037] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:04.143916 21611 raft_consensus.cc:504] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.144541 21611 leader_election.cc:288] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [CANDIDATE]: Term 1 election: Requested vote from peers 28079dd3892345c1afab5fc32aeacf2c (127.2.205.131:46069), 31ddfa83759f4646be74be623c691bf7 (127.2.205.129:40535)
I20260929 03:06:04.145246 21273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa13f375a49431fa1c4e4ea5f22097d" candidate_uuid: "bebe105067264ce0a9fe9870a3c85815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ddfa83759f4646be74be623c691bf7"
I20260929 03:06:04.145221 21541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa13f375a49431fa1c4e4ea5f22097d" candidate_uuid: "bebe105067264ce0a9fe9870a3c85815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28079dd3892345c1afab5fc32aeacf2c"
I20260929 03:06:04.145462 21273 raft_consensus.cc:3037] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:04.145462 21541 raft_consensus.cc:3037] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:04.148775 21541 raft_consensus.cc:2454] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebe105067264ce0a9fe9870a3c85815 in term 1.
I20260929 03:06:04.148775 21273 raft_consensus.cc:2454] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebe105067264ce0a9fe9870a3c85815 in term 1.
I20260929 03:06:04.149339 21340 leader_election.cc:302] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28079dd3892345c1afab5fc32aeacf2c, 31ddfa83759f4646be74be623c691bf7, bebe105067264ce0a9fe9870a3c85815; no voters:
I20260929 03:06:04.149729 21611 raft_consensus.cc:2781] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:04.150171 21611 raft_consensus.cc:686] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 1 LEADER]: Becoming Leader. State: Replica: bebe105067264ce0a9fe9870a3c85815, State: Running, Role: LEADER
I20260929 03:06:04.150677 21611 consensus_queue.cc:237] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } }
I20260929 03:06:04.155241 21128 catalog_manager.cc:5495] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 reported cstate change: term changed from 0 to 1, leader changed from <none> to bebe105067264ce0a9fe9870a3c85815 (127.2.205.130). New cstate: current_term: 1 leader_uuid: "bebe105067264ce0a9fe9870a3c85815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebe105067264ce0a9fe9870a3c85815" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 33695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 } health_report { overall_health: UNKNOWN } } }
I20260929 03:06:04.178933 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.172713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.2.205.130:33695)
0929 03:06:04.172969 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.172974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.172994 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:06:04.173107 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.173113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.173206 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.173331 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:04.173789 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.174688 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.176159 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.176815 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.176937 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.177522 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:04.178209 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:04.178378 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.178581 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.178633 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 03:06:04.187963 21273 raft_consensus.cc:1261] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 1 FOLLOWER]: Refusing update from remote peer bebe105067264ce0a9fe9870a3c85815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:04.187963 21541 raft_consensus.cc:1261] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 1 FOLLOWER]: Refusing update from remote peer bebe105067264ce0a9fe9870a3c85815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:04.189239 21616 consensus_queue.cc:1035] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ddfa83759f4646be74be623c691bf7" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:04.189689 21611 consensus_queue.cc:1035] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28079dd3892345c1afab5fc32aeacf2c" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:04.201784 21619 mvcc.cc:205] Tried to move back new op lower bound from 7334507168501923840 to 7334507168365293568. Current Snapshot: MvccSnapshot[applied={T|T < 7334507168501923840}]
I20260929 03:06:04.214637 21128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:04.214860 21128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:04.218391 21128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96288723d83a44f4872ef233ea45581b: Sending DeleteTablet for 3 replicas of tablet daa13f375a49431fa1c4e4ea5f22097d
I20260929 03:06:04.219534 21387 tablet_service.cc:1514] Processing DeleteTablet for tablet daa13f375a49431fa1c4e4ea5f22097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:04 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:45508
I20260929 03:06:04.219628 21521 tablet_service.cc:1514] Processing DeleteTablet for tablet daa13f375a49431fa1c4e4ea5f22097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:04 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:40796
I20260929 03:06:04.219705 21253 tablet_service.cc:1514] Processing DeleteTablet for tablet daa13f375a49431fa1c4e4ea5f22097d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:04 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:53530
I20260929 03:06:04.229404 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21190
I20260929 03:06:04.232402 21631 tablet_replica.cc:330] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: stopping tablet replica
I20260929 03:06:04.233152 21631 raft_consensus.cc:2229] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:04.233868 21631 raft_consensus.cc:2258] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:04.234256 21629 tablet_replica.cc:330] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: stopping tablet replica
I20260929 03:06:04.234812 21630 tablet_replica.cc:330] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: stopping tablet replica
I20260929 03:06:04.235309 21630 raft_consensus.cc:2229] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:04.235317 21629 raft_consensus.cc:2229] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:06:04.235870 21630 raft_consensus.cc:2258] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:04.235991 21629 raft_consensus.cc:2258] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:04.236368 21631 ts_tablet_manager.cc:1905] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:04.238194 21630 ts_tablet_manager.cc:1905] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:04.243422 21629 ts_tablet_manager.cc:1905] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:04.243954 21631 ts_tablet_manager.cc:1918] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:04.244141 21631 log.cc:1196] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/daa13f375a49431fa1c4e4ea5f22097d
I20260929 03:06:04.244695 21631 ts_tablet_manager.cc:1939] T daa13f375a49431fa1c4e4ea5f22097d P 31ddfa83759f4646be74be623c691bf7: Deleting consensus metadata
I20260929 03:06:04.246783 21116 catalog_manager.cc:4841] TS 31ddfa83759f4646be74be623c691bf7 (127.2.205.129:40535): tablet daa13f375a49431fa1c4e4ea5f22097d (table test-table [id=b493188833074d30939a7e577fd91361]) successfully deleted
Sep 29 03:06:04 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:06:04.249679 21630 ts_tablet_manager.cc:1918] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:04.249858 21630 log.cc:1196] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/daa13f375a49431fa1c4e4ea5f22097d
I20260929 03:06:04.250362 21630 ts_tablet_manager.cc:1939] T daa13f375a49431fa1c4e4ea5f22097d P 28079dd3892345c1afab5fc32aeacf2c: Deleting consensus metadata
I20260929 03:06:04.252195 21113 catalog_manager.cc:4841] TS 28079dd3892345c1afab5fc32aeacf2c (127.2.205.131:46069): tablet daa13f375a49431fa1c4e4ea5f22097d (table test-table [id=b493188833074d30939a7e577fd91361]) successfully deleted
I20260929 03:06:04.253633 21629 ts_tablet_manager.cc:1918] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:04.253809 21629 log.cc:1196] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/daa13f375a49431fa1c4e4ea5f22097d
I20260929 03:06:04.254298 21629 ts_tablet_manager.cc:1939] T daa13f375a49431fa1c4e4ea5f22097d P bebe105067264ce0a9fe9870a3c85815: Deleting consensus metadata
I20260929 03:06:04.256017 21114 catalog_manager.cc:4841] TS bebe105067264ce0a9fe9870a3c85815 (127.2.205.130:33695): tablet daa13f375a49431fa1c4e4ea5f22097d (table test-table [id=b493188833074d30939a7e577fd91361]) successfully deleted
I20260929 03:06:04.256083 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.231466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.2.205.129:40535)
0929 03:06:04.232439 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.232445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.232470 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:06:04.232539 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.232544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.232629 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.232740 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:04.237205 (+ 4465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.238350 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.240267 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.240903 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.241011 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.248434 (+ 7423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:04.248450 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:04.248468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:04.248525 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:04.252490 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.253417 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.253429 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.253432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.253613 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.254088 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.254093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.254097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.254570 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:04.254871 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.255256 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.255491 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829}
I20260929 03:06:04.256778 21313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:04.382210 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21190
I20260929 03:06:04.396852 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21324
Sep 29 03:06:04 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:06:04.413354 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.397685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.2.205.130:33695)
0929 03:06:04.397863 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.397868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.397890 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:06:04.398053 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.398065 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.398135 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.398216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:04.399595 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.400488 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.402037 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.402485 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.402599 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.406859 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:04.406876 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:04.406892 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:04.406929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:04.410176 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.410933 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.410941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.410944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.411111 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.411500 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.411504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.411510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.411909 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:04.412273 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.412587 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.412875 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:04.414160 21447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:04.530704 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21324
I20260929 03:06:04.546620 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21458
Sep 29 03:06:04 dist-test-slave-530n krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:06:04.561244 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:04.547439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.2.205.131:46069)
0929 03:06:04.547632 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:04.547637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:04.547658 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:06:04.547757 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:04.547762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:04.547835 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:04.547918 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:04.549252 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.549951 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.551176 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:04.551554 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:04.551658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:04.555852 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:04.555866 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:04.555882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:04.555915 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:04.558585 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.559245 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.559252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.559254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.559392 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:04.559757 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:04.559760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:04.559762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:04.560112 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:04.560371 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:04.560681 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:06:04.560909 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:04.561869 21581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:04.650607 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21458
I20260929 03:06:04.663646 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21097
I20260929 03:06:04.664700 21159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:04.776827 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21097
2026-09-29T03:06:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4304 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:04Z Disabled control of system clock
I20260929 03:06:04.849201 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:44043
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:44043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:06:05.180516 21640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:05.180898 21640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:05.181048 21640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:05.188906 21640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:05.189052 21640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:05.189091 21640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:05.189169 21640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:05.189230 21640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:05.199067 21640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:44043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:44043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:05.200527 21640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:05.202371 21640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:05.213788 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:05.213801 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:05.213835 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:05.214607 21640 server_base.cc:1034] running on GCE node
I20260929 03:06:05.215335 21640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:05.216723 21640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:05.217952 21640 hybrid_clock.cc:648] HybridClock initialized: now 1790651165217881 us; error 59 us; skew 500 ppm
I20260929 03:06:05.218290 21640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:05.220479 21640 webserver.cc:460] Webserver started at http://127.2.205.190:45213/ using document root <none> and password file <none>
I20260929 03:06:05.221228 21640 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:05.221314 21640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:05.221621 21640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:05.224438 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "6ddf51165c134ca2aaf5d4315a5b1ead"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "c45195aa0fc3971399c9e7110bd715ea"
server_key_iv: "fd3318c59b0755c2ee1d6653ed9625ab"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.225232 21640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "6ddf51165c134ca2aaf5d4315a5b1ead"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "c45195aa0fc3971399c9e7110bd715ea"
server_key_iv: "fd3318c59b0755c2ee1d6653ed9625ab"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.230501 21640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:06:05.234172 21654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:05.235936 21640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:05.236138 21640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "6ddf51165c134ca2aaf5d4315a5b1ead"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "c45195aa0fc3971399c9e7110bd715ea"
server_key_iv: "fd3318c59b0755c2ee1d6653ed9625ab"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.236397 21640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:05.256549 21640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:05.257500 21640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:05.257782 21640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:05.275748 21640 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:44043
I20260929 03:06:05.275770 21705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:44043 every 8 connection(s)
I20260929 03:06:05.277155 21640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:05.281138 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:05.281790 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21640
I20260929 03:06:05.281963 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:05.282420 2870 external_mini_cluster.cc:1428] Setting key ee7bbf8025e9bd39b3e3cd3b21fd3fc0
I20260929 03:06:05.296168 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: Bootstrap starting.
I20260929 03:06:05.298700 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:05.284440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.2.205.190:44043)
0929 03:06:05.285621 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:05.285639 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:05.285706 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:06:05.287929 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:05.287966 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:05.288798 (+ 832us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:05.289350 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:05.289916 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.291830 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.293333 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.294406 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.294581 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:05.295702 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:05.295712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:05.295735 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:05.296465 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:05.296581 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:05.296609 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:05.296721 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:05.296967 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 03:06:05.297379 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":342,"threads_started":1}
I20260929 03:06:05.300374 21706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:05.301496 21706 log.cc:824] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:05.304289 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: No bootstrap required, opened a new log
I20260929 03:06:05.312062 21706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } }
I20260929 03:06:05.312474 21706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:05.312600 21706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ddf51165c134ca2aaf5d4315a5b1ead, State: Initialized, Role: FOLLOWER
I20260929 03:06:05.313257 21706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } }
I20260929 03:06:05.313482 21706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:05.313624 21706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:05.313786 21706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:05.315999 21706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } }
I20260929 03:06:05.316551 21706 leader_election.cc:302] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ddf51165c134ca2aaf5d4315a5b1ead; no voters:
I20260929 03:06:05.317291 21706 leader_election.cc:288] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:05.317446 21711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:05.318202 21711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [term 1 LEADER]: Becoming Leader. State: Replica: 6ddf51165c134ca2aaf5d4315a5b1ead, State: Running, Role: LEADER
I20260929 03:06:05.318761 21711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } }
I20260929 03:06:05.319482 21706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:05.321872 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } } }
I20260929 03:06:05.321869 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ddf51165c134ca2aaf5d4315a5b1ead. Latest consensus state: current_term: 1 leader_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ddf51165c134ca2aaf5d4315a5b1ead" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 44043 } } }
I20260929 03:06:05.322336 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:05.322428 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:05.323870 21717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:05.329243 21717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:05.340637 21717 catalog_manager.cc:1347] Generated new cluster ID: 2d33c3290b3b47d3a39097b8630e7cd7
I20260929 03:06:05.340782 21717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:05.354672 21717 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:05.356216 21717 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:05.364899 21717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: Generated new TSK 0
I20260929 03:06:05.365723 21717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:05.371474 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--builtin_ntp_servers=127.2.205.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:06:05.719525 21730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:05.719851 21730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:05.719985 21730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:05.727538 21730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:05.727659 21730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:05.727766 21730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:06:05.737268 21730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:05.738504 21730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:05.740271 21730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:05.756701 21735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:05.756768 21736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:05.756817 21738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:05.758080 21730 server_base.cc:1034] running on GCE node
I20260929 03:06:05.758668 21730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:05.759616 21730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:05.760869 21730 hybrid_clock.cc:648] HybridClock initialized: now 1790651165760820 us; error 34 us; skew 500 ppm
I20260929 03:06:05.761222 21730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:05.763445 21730 webserver.cc:460] Webserver started at http://127.2.205.129:38983/ using document root <none> and password file <none>
I20260929 03:06:05.764137 21730 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:05.764240 21730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:05.764544 21730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:05.767326 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "cd9ef37ea6254595b0c6d80849245b48"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "8e2b1c81eaa0da0d233bf9628c2d380b"
server_key_iv: "68ef4cd3c3f8d81aaebf7f4875bd0f5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.768069 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "cd9ef37ea6254595b0c6d80849245b48"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "8e2b1c81eaa0da0d233bf9628c2d380b"
server_key_iv: "68ef4cd3c3f8d81aaebf7f4875bd0f5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.772888 21730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 03:06:05.776057 21744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:05.777369 21730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:05.777518 21730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "cd9ef37ea6254595b0c6d80849245b48"
format_stamp: "Formatted at 2026-09-29 03:06:05 on dist-test-slave-530n"
server_key: "8e2b1c81eaa0da0d233bf9628c2d380b"
server_key_iv: "68ef4cd3c3f8d81aaebf7f4875bd0f5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:05.777702 21730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:05.796226 21730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:05.797185 21730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:05.797485 21730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:05.798513 21730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:05.800038 21730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:05.800143 21730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:05.800264 21730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:05.800349 21730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:05.831761 21730 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:41591
I20260929 03:06:05.831813 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:41591 every 8 connection(s)
I20260929 03:06:05.833214 21730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:06:05.833652 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21730
I20260929 03:06:05.833814 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:06:05.834412 2870 external_mini_cluster.cc:1428] Setting key a40136abc08af0270911d348a6071221
I20260929 03:06:05.837783 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--builtin_ntp_servers=127.2.205.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:06:05.846949 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:05.836133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:43337 (local address 127.2.205.190:44043)
0929 03:06:05.836742 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:05.836771 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:05.838211 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 03:06:05.840104 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:05.840110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:05.840207 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:05.840311 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:05.841547 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.842312 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.844043 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.844541 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.844667 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:05.846141 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:05.846148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:05.846153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:05.846185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:05.846230 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:05.846245 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:05.846334 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:05.846520 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:06:05.846621 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":409,"threads_started":1}
I20260929 03:06:05.847891 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:05.836959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:44043 (local address 127.2.205.129:43337)
0929 03:06:05.838031 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:05.838085 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:05.839822 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:05.840590 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:05.840600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:05.840706 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:05.841370 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:05.841388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.842470 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.842474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:05.843888 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:05.843901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:05.844832 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:05.844841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:05.845137 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:05.845806 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:05.845809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:05.845849 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:05.845940 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:05.846334 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:05.846341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:05.846346 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:06:05.846480 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 03:06:05.846822 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":305,"threads_started":1}
I20260929 03:06:05.849463 21857 heartbeater.cc:347] Connected to a master server at 127.2.205.190:44043
I20260929 03:06:05.849757 21857 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:05.850478 21857 heartbeater.cc:511] Master 127.2.205.190:44043 requested a full tablet report, sending...
I20260929 03:06:05.852478 21671 ts_manager.cc:194] Registered new tserver with Master: cd9ef37ea6254595b0c6d80849245b48 (127.2.205.129:41591)
I20260929 03:06:05.854013 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.129:43337
W20260929 03:06:06.163132 21861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:06.163429 21861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:06.163537 21861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:06.170971 21861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:06.171090 21861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:06.171228 21861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:06:06.180495 21861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:06.181883 21861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:06.183840 21861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:06.197311 21867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:06.197275 21869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:06.197460 21866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:06.198285 21861 server_base.cc:1034] running on GCE node
I20260929 03:06:06.198910 21861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:06.199918 21861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:06.201122 21861 hybrid_clock.cc:648] HybridClock initialized: now 1790651166201076 us; error 57 us; skew 500 ppm
I20260929 03:06:06.201433 21861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:06.203697 21861 webserver.cc:460] Webserver started at http://127.2.205.130:45197/ using document root <none> and password file <none>
I20260929 03:06:06.204377 21861 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:06.204480 21861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:06.204852 21861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:06.207863 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "2ea134acbb184666b9a010bfcd3f5ede"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "a500abaa21d2bcd0e88b44d6f6fb7ae4"
server_key_iv: "e29cc1fb4e21334adfaf5df6d5b12046"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.208711 21861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "2ea134acbb184666b9a010bfcd3f5ede"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "a500abaa21d2bcd0e88b44d6f6fb7ae4"
server_key_iv: "e29cc1fb4e21334adfaf5df6d5b12046"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.214241 21861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:06:06.218111 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.219702 21861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 03:06:06.219882 21861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "2ea134acbb184666b9a010bfcd3f5ede"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "a500abaa21d2bcd0e88b44d6f6fb7ae4"
server_key_iv: "e29cc1fb4e21334adfaf5df6d5b12046"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.220119 21861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:06.249927 21861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:06.250808 21861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:06.251094 21861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:06.252137 21861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:06.253633 21861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:06.253727 21861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.253839 21861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:06.253916 21861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.286273 21861 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:38929
I20260929 03:06:06.286319 21987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:38929 every 8 connection(s)
I20260929 03:06:06.287652 21861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:06:06.292800 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21861
I20260929 03:06:06.293007 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:06:06.293661 2870 external_mini_cluster.cc:1428] Setting key 8f2a81800bf896fac2a16efcdcd150ce
I20260929 03:06:06.297008 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--builtin_ntp_servers=127.2.205.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:06:06.302428 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.290573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:40073 (local address 127.2.205.190:44043)
0929 03:06:06.290749 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.290754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.292501 (+ 1747us) server_negotiation.cc:407] Connection header received
0929 03:06:06.295176 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.295181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.295254 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.295355 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:06.296783 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.297550 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.299720 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.300118 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.300212 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.301720 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:06.301726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:06.301729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:06.301750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:06.301780 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:06.301793 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:06.301859 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.302097 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.302188 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:06.303882 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.291226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:44043 (local address 127.2.205.130:40073)
0929 03:06:06.292308 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:06.292373 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:06.294834 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:06.295633 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:06.295645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:06.295767 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:06.296570 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.296600 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.297714 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.297720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.299558 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.299573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.300332 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.300337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.300615 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.301342 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:06.301346 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:06.301407 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:06.301518 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:06.301879 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:06.301887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:06.301895 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:06:06.302084 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 03:06:06.302534 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":322,"threads_started":1}
I20260929 03:06:06.305387 21988 heartbeater.cc:347] Connected to a master server at 127.2.205.190:44043
I20260929 03:06:06.305694 21988 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:06.306478 21988 heartbeater.cc:511] Master 127.2.205.190:44043 requested a full tablet report, sending...
I20260929 03:06:06.308131 21671 ts_manager.cc:194] Registered new tserver with Master: 2ea134acbb184666b9a010bfcd3f5ede (127.2.205.130:38929)
I20260929 03:06:06.309238 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.130:40073
W20260929 03:06:06.636170 21991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:06.636474 21991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:06.636561 21991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:06.644145 21991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:06.644269 21991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:06.644380 21991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:06:06.653746 21991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:44043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:06.655047 21991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:06.656713 21991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:06.668830 21996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:06.668800 21999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:06.668851 21997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:06.669610 21991 server_base.cc:1034] running on GCE node
I20260929 03:06:06.670130 21991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:06.670965 21991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:06.672175 21991 hybrid_clock.cc:648] HybridClock initialized: now 1790651166672120 us; error 42 us; skew 500 ppm
I20260929 03:06:06.672475 21991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:06.674741 21991 webserver.cc:460] Webserver started at http://127.2.205.131:40711/ using document root <none> and password file <none>
I20260929 03:06:06.675380 21991 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:06.675467 21991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:06.675760 21991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:06.678679 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "9d7847308406483a8d463f8a994060f3"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "11f913c7cac2451e325677dc7b5b4796"
server_key_iv: "b7db8033bbb9447db34a3011570205fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.679503 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "9d7847308406483a8d463f8a994060f3"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "11f913c7cac2451e325677dc7b5b4796"
server_key_iv: "b7db8033bbb9447db34a3011570205fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.685190 21991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:06:06.688810 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.690236 21991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:06.690397 21991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "9d7847308406483a8d463f8a994060f3"
format_stamp: "Formatted at 2026-09-29 03:06:06 on dist-test-slave-530n"
server_key: "11f913c7cac2451e325677dc7b5b4796"
server_key_iv: "b7db8033bbb9447db34a3011570205fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:06.690616 21991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:06.710639 21991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:06.711540 21991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:06.711846 21991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:06.712920 21991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:06.714499 21991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:06.714591 21991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.714704 21991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:06.714776 21991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:06.745272 21991 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:34871
I20260929 03:06:06.745327 22117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:34871 every 8 connection(s)
I20260929 03:06:06.746578 21991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:06:06.747517 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 21991
I20260929 03:06:06.747707 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:06:06.748240 2870 external_mini_cluster.cc:1428] Setting key 3bd339ede0e86f34187c5df651716dbc
I20260929 03:06:06.759989 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.749540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:48145 (local address 127.2.205.190:44043)
0929 03:06:06.749737 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.749743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.751795 (+ 2052us) server_negotiation.cc:407] Connection header received
0929 03:06:06.753627 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.753631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.753700 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.753780 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:06.754874 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.755617 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.757395 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.757843 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.757949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.759331 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:06.759336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:06.759339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:06.759358 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:06.759394 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:06.759408 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:06.759478 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.759658 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.759742 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:06:06.761082 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.750354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:44043 (local address 127.2.205.131:48145)
0929 03:06:06.751609 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:06.751661 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:06.753380 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:06.754007 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:06.754017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:06.754123 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:06.754717 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.754734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.755790 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.755794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.757230 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.757244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.758102 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.758108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.758328 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.758980 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:06.758982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:06.759021 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:06.759132 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:06.759485 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:06.759491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:06.759496 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:06:06.759635 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 03:06:06.759940 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":366,"threads_started":1}
I20260929 03:06:06.762498 22118 heartbeater.cc:347] Connected to a master server at 127.2.205.190:44043
I20260929 03:06:06.762811 22118 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:06.763661 22118 heartbeater.cc:511] Master 127.2.205.190:44043 requested a full tablet report, sending...
I20260929 03:06:06.765084 21671 ts_manager.cc:194] Registered new tserver with Master: 9d7847308406483a8d463f8a994060f3 (127.2.205.131:34871)
I20260929 03:06:06.765872 21671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.131:48145
I20260929 03:06:06.777462 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:06:06.787829 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.781716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.2.205.190:44043)
0929 03:06:06.781891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.781897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.782371 (+ 474us) server_negotiation.cc:407] Connection header received
0929 03:06:06.782551 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.782555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.782637 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.782723 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:06.783258 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.783978 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.785412 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.785818 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.785928 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.787119 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:06.787124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:06.787128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:06.787147 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:06.787179 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:06.787191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:06.787318 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.787514 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.787590 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:06:06.791872 21671 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:06:06.799166 21671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:06:06.833385 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.811709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:38929 (local address 127.0.0.1:45210)
0929 03:06:06.812834 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:06.812876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:06.813067 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:06.825526 (+ 12459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:06.825535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:06.825566 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:06.826091 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.826110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.828321 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.828326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.830618 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.830629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.832490 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.832497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.832668 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.832701 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:06:06.832795 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:06:06.832933 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":165,"thread_start_us":1302,"threads_started":1}
I20260929 03:06:06.833576 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.814015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36420)
0929 03:06:06.825159 (+ 11144us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:06.825178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:06.825300 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:06.826165 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:06.826169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:06.826192 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:06.826556 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.826568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.828373 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.828377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.830600 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.830614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.832793 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.832799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.832989 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.833052 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 03:06:06.833126 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:06:06.833237 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11025,"mutex_wait_us":155,"thread_start_us":10886,"threads_started":1}
I20260929 03:06:06.834400 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.814587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.2.205.129:41591)
0929 03:06:06.824727 (+ 10140us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.824735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.825422 (+ 687us) server_negotiation.cc:407] Connection header received
0929 03:06:06.825522 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.825540 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.825754 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.825942 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:06.826753 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.828226 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.830825 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.832271 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.832402 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.832651 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.833761 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.833953 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9967,"thread_start_us":9828,"threads_started":1}
I20260929 03:06:06.834435 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.812030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:34871 (local address 127.0.0.1:40770)
0929 03:06:06.824379 (+ 12349us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:06.824402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:06.824574 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:06.825765 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:06.825769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:06.825792 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:06.826269 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.826284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.827793 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.827797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.830122 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:06.830137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.832266 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.832272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:06.832475 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.832535 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 03:06:06.832687 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:06:06.832946 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12213,"mutex_wait_us":1,"thread_start_us":12008,"threads_started":1}
I20260929 03:06:06.834957 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.811978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.2.205.130:38929)
0929 03:06:06.824826 (+ 12848us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.824834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.824866 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:06:06.824986 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.824993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.825269 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.826430 (+ 1161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:06.826633 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.828165 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.830834 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.832271 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.833686 (+ 1415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.833910 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.833979 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.834700 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12700,"thread_start_us":12438,"threads_started":1}
I20260929 03:06:06.835103 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:06.812594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.2.205.131:34871)
0929 03:06:06.823634 (+ 11040us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:06.823645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:06.824736 (+ 1091us) server_negotiation.cc:407] Connection header received
0929 03:06:06.824888 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:06.824899 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:06.825279 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:06.825480 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:06.826506 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.827643 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.830317 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:06.831746 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:06.831869 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:06.832144 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:06.833065 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 03:06:06.834783 (+ 1718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10815,"thread_start_us":11074,"threads_started":1}
I20260929 03:06:06.839409 21792 tablet_service.cc:1467] Processing CreateTablet for tablet da6d15d612834a919ce7468394c48eba (DEFAULT_TABLE table=test-table [id=557517534d0f41dba7d31996e0b8da38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:06.839409 22053 tablet_service.cc:1467] Processing CreateTablet for tablet da6d15d612834a919ce7468394c48eba (DEFAULT_TABLE table=test-table [id=557517534d0f41dba7d31996e0b8da38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:06.839622 21923 tablet_service.cc:1467] Processing CreateTablet for tablet da6d15d612834a919ce7468394c48eba (DEFAULT_TABLE table=test-table [id=557517534d0f41dba7d31996e0b8da38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:06.841094 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d15d612834a919ce7468394c48eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:06.841120 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d15d612834a919ce7468394c48eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:06.841109 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d15d612834a919ce7468394c48eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:06.853308 22136 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Bootstrap starting.
I20260929 03:06:06.856112 22138 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Bootstrap starting.
I20260929 03:06:06.856878 21857 heartbeater.cc:503] Master 127.2.205.190:44043 was elected leader, sending a full tablet report...
I20260929 03:06:06.859215 22136 tablet_bootstrap.cc:654] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:06.860534 22136 log.cc:824] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:06.861449 22137 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Bootstrap starting.
I20260929 03:06:06.861989 22138 tablet_bootstrap.cc:654] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:06.863148 22138 log.cc:824] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:06.864880 22136 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: No bootstrap required, opened a new log
I20260929 03:06:06.865178 22136 ts_tablet_manager.cc:1397] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 03:06:06.865662 22137 tablet_bootstrap.cc:654] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:06.866564 22137 log.cc:824] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:06.867280 22138 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: No bootstrap required, opened a new log
I20260929 03:06:06.867530 22138 ts_tablet_manager.cc:1397] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260929 03:06:06.868875 22137 tablet_bootstrap.cc:492] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: No bootstrap required, opened a new log
I20260929 03:06:06.869112 22137 ts_tablet_manager.cc:1397] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 03:06:06.875219 22136 raft_consensus.cc:348] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.875669 22136 raft_consensus.cc:374] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:06.875805 22136 raft_consensus.cc:729] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea134acbb184666b9a010bfcd3f5ede, State: Initialized, Role: FOLLOWER
I20260929 03:06:06.876121 22137 raft_consensus.cc:348] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.876458 22137 raft_consensus.cc:374] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:06.876552 22137 raft_consensus.cc:729] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9ef37ea6254595b0c6d80849245b48, State: Initialized, Role: FOLLOWER
I20260929 03:06:06.876411 22136 consensus_queue.cc:260] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.877081 22137 consensus_queue.cc:260] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.877933 22138 raft_consensus.cc:348] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.878435 22138 raft_consensus.cc:374] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:06.878562 22138 raft_consensus.cc:729] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7847308406483a8d463f8a994060f3, State: Initialized, Role: FOLLOWER
I20260929 03:06:06.878743 22137 ts_tablet_manager.cc:1428] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260929 03:06:06.879341 22138 consensus_queue.cc:260] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:06.880517 22118 heartbeater.cc:503] Master 127.2.205.190:44043 was elected leader, sending a full tablet report...
I20260929 03:06:06.881260 22138 ts_tablet_manager.cc:1428] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260929 03:06:06.882098 21988 heartbeater.cc:503] Master 127.2.205.190:44043 was elected leader, sending a full tablet report...
I20260929 03:06:06.882840 22136 ts_tablet_manager.cc:1428] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260929 03:06:06.998270 22119 tablet.cc:2369] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:07.027650 22143 raft_consensus.cc:482] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:06:07.027959 22143 raft_consensus.cc:504] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:07.029764 22143 leader_election.cc:288] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ea134acbb184666b9a010bfcd3f5ede (127.2.205.130:38929), 9d7847308406483a8d463f8a994060f3 (127.2.205.131:34871)
I20260929 03:06:07.038863 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.030208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:41385 (local address 127.2.205.130:38929)
0929 03:06:07.030424 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.030431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.031214 (+ 783us) server_negotiation.cc:407] Connection header received
0929 03:06:07.031356 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.031360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.031443 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.031534 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:07.032260 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.033429 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.036558 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.037775 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.037945 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.038018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.038457 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.038599 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:06:07.038869 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.030418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:47713 (local address 127.2.205.131:34871)
0929 03:06:07.030622 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.030628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.031067 (+ 439us) server_negotiation.cc:407] Connection header received
0929 03:06:07.031320 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.031326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.031401 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.031495 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:07.032296 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.033440 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.036551 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.037789 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.037951 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.038021 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.038461 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.038599 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:06:07.038993 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.030174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:38929 (local address 127.2.205.129:41385)
0929 03:06:07.031106 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.031120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:07.031230 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:07.031550 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:07.031553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:07.031570 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:07.032049 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:07.032066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.033596 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.033601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:07.036297 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:07.036315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.037968 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.037975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:07.038218 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.038317 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 03:06:07.038409 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:06:07.038593 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":291,"thread_start_us":626,"threads_started":1}
I20260929 03:06:07.039260 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.030125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:34871 (local address 127.2.205.129:47713)
0929 03:06:07.030936 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.030951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:07.031115 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:07.031532 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:07.031537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:07.031562 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:07.032140 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:07.032151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.033599 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.033603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:07.036296 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:07.036315 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.038003 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.038010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:07.038222 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.038318 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:06:07.038409 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:06:07.038593 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":355,"thread_start_us":498,"threads_started":1}
I20260929 03:06:07.040211 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d15d612834a919ce7468394c48eba" candidate_uuid: "cd9ef37ea6254595b0c6d80849245b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea134acbb184666b9a010bfcd3f5ede" is_pre_election: true
I20260929 03:06:07.040299 22073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d15d612834a919ce7468394c48eba" candidate_uuid: "cd9ef37ea6254595b0c6d80849245b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7847308406483a8d463f8a994060f3" is_pre_election: true
I20260929 03:06:07.040863 22073 raft_consensus.cc:2454] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9ef37ea6254595b0c6d80849245b48 in term 0.
I20260929 03:06:07.040868 21943 raft_consensus.cc:2454] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd9ef37ea6254595b0c6d80849245b48 in term 0.
W20260929 03:06:07.041086 21989 tablet.cc:2369] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:07.041605 21745 leader_election.cc:302] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea134acbb184666b9a010bfcd3f5ede, cd9ef37ea6254595b0c6d80849245b48; no voters:
I20260929 03:06:07.042053 22143 raft_consensus.cc:2781] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:06:07.042208 22143 raft_consensus.cc:482] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:06:07.042308 22143 raft_consensus.cc:3037] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:07.044411 22143 raft_consensus.cc:504] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:07.045063 22143 leader_election.cc:288] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [CANDIDATE]: Term 1 election: Requested vote from peers 2ea134acbb184666b9a010bfcd3f5ede (127.2.205.130:38929), 9d7847308406483a8d463f8a994060f3 (127.2.205.131:34871)
I20260929 03:06:07.045490 21943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d15d612834a919ce7468394c48eba" candidate_uuid: "cd9ef37ea6254595b0c6d80849245b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea134acbb184666b9a010bfcd3f5ede"
I20260929 03:06:07.045677 21943 raft_consensus.cc:3037] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:07.045615 22073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d15d612834a919ce7468394c48eba" candidate_uuid: "cd9ef37ea6254595b0c6d80849245b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7847308406483a8d463f8a994060f3"
I20260929 03:06:07.045820 22073 raft_consensus.cc:3037] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:07.047957 21943 raft_consensus.cc:2454] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9ef37ea6254595b0c6d80849245b48 in term 1.
I20260929 03:06:07.047956 22073 raft_consensus.cc:2454] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd9ef37ea6254595b0c6d80849245b48 in term 1.
I20260929 03:06:07.048339 21747 leader_election.cc:302] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7847308406483a8d463f8a994060f3, cd9ef37ea6254595b0c6d80849245b48; no voters:
I20260929 03:06:07.048640 22143 raft_consensus.cc:2781] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:07.049077 22143 raft_consensus.cc:686] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 1 LEADER]: Becoming Leader. State: Replica: cd9ef37ea6254595b0c6d80849245b48, State: Running, Role: LEADER
I20260929 03:06:07.049574 22143 consensus_queue.cc:237] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } }
I20260929 03:06:07.053800 21670 catalog_manager.cc:5495] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd9ef37ea6254595b0c6d80849245b48 (127.2.205.129). New cstate: current_term: 1 leader_uuid: "cd9ef37ea6254595b0c6d80849245b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd9ef37ea6254595b0c6d80849245b48" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 41591 } health_report { overall_health: HEALTHY } } }
I20260929 03:06:07.073390 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.067851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.2.205.129:41591)
0929 03:06:07.068082 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.068087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.068135 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:06:07.068354 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.068359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.068446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.068584 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:07.069019 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.069693 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.071042 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.071454 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.071573 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.072258 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:07.072791 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:07.072986 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.073106 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.073145 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:06:07.081453 22073 raft_consensus.cc:1261] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 1 FOLLOWER]: Refusing update from remote peer cd9ef37ea6254595b0c6d80849245b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:07.081452 21943 raft_consensus.cc:1261] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 1 FOLLOWER]: Refusing update from remote peer cd9ef37ea6254595b0c6d80849245b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:07.082463 22143 consensus_queue.cc:1035] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea134acbb184666b9a010bfcd3f5ede" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:07.082801 22147 consensus_queue.cc:1035] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7847308406483a8d463f8a994060f3" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 34871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:06:07.086642 21858 tablet.cc:2369] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:07.111112 21671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:07.111469 21671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:07.115775 21671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ddf51165c134ca2aaf5d4315a5b1ead: Sending DeleteTablet for 3 replicas of tablet da6d15d612834a919ce7468394c48eba
I20260929 03:06:07.117188 22053 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d15d612834a919ce7468394c48eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:07 UTC) from {username='slave'} at 127.0.0.1:40770
I20260929 03:06:07.117182 21923 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d15d612834a919ce7468394c48eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:07 UTC) from {username='slave'} at 127.0.0.1:45210
I20260929 03:06:07.117574 21792 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d15d612834a919ce7468394c48eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:07 UTC) from {username='slave'} at 127.0.0.1:36420
I20260929 03:06:07.118918 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21730
I20260929 03:06:07.129782 22161 tablet_replica.cc:330] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: stopping tablet replica
I20260929 03:06:07.130542 22162 tablet_replica.cc:330] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: stopping tablet replica
I20260929 03:06:07.131358 22162 raft_consensus.cc:2229] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:06:07.132025 22161 raft_consensus.cc:2229] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:07.133371 22162 raft_consensus.cc:2258] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:07.133661 22161 raft_consensus.cc:2258] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:07.136649 22161 ts_tablet_manager.cc:1905] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:07.138950 22162 ts_tablet_manager.cc:1905] T da6d15d612834a919ce7468394c48eba P cd9ef37ea6254595b0c6d80849245b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:07.139627 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.120483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.2.205.129:41591)
0929 03:06:07.120695 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.120701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.130823 (+ 10122us) server_negotiation.cc:407] Connection header received
0929 03:06:07.131160 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.131167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.131284 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.131373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:07.132095 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.133398 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.135415 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.136121 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.136228 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.137771 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:07.137783 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:07.137801 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:07.138628 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:07.138708 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:07.138734 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:07.138824 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.139093 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.139216 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:06:07.140942 21852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:07.141603 22160 tablet_replica.cc:330] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: stopping tablet replica
I20260929 03:06:07.142216 22160 raft_consensus.cc:2229] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:07.143003 22160 raft_consensus.cc:2258] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:07.145424 22160 ts_tablet_manager.cc:1905] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:07.145597 22161 ts_tablet_manager.cc:1918] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:07.145773 22161 log.cc:1196] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/da6d15d612834a919ce7468394c48eba
I20260929 03:06:07.146404 22161 ts_tablet_manager.cc:1939] T da6d15d612834a919ce7468394c48eba P 2ea134acbb184666b9a010bfcd3f5ede: Deleting consensus metadata
I20260929 03:06:07.148499 21655 catalog_manager.cc:4841] TS 2ea134acbb184666b9a010bfcd3f5ede (127.2.205.130:38929): tablet da6d15d612834a919ce7468394c48eba (table test-table [id=557517534d0f41dba7d31996e0b8da38]) successfully deleted
I20260929 03:06:07.151271 22160 ts_tablet_manager.cc:1918] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:07.151407 22160 log.cc:1196] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/da6d15d612834a919ce7468394c48eba
I20260929 03:06:07.151814 22160 ts_tablet_manager.cc:1939] T da6d15d612834a919ce7468394c48eba P 9d7847308406483a8d463f8a994060f3: Deleting consensus metadata
I20260929 03:06:07.153405 21657 catalog_manager.cc:4841] TS 9d7847308406483a8d463f8a994060f3 (127.2.205.131:34871): tablet da6d15d612834a919ce7468394c48eba (table test-table [id=557517534d0f41dba7d31996e0b8da38]) successfully deleted
I20260929 03:06:07.241317 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21730
I20260929 03:06:07.257377 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21861
W20260929 03:06:07.257440 21658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:41591 (error 108)
I20260929 03:06:07.259362 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.258883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36438)
0929 03:06:07.259070 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.259191 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 03:06:07.259598 21658 catalog_manager.cc:4584] TS cd9ef37ea6254595b0c6d80849245b48 (127.2.205.129:41591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da6d15d612834a919ce7468394c48eba: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
I20260929 03:06:07.264262 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.258174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.2.205.130:38929)
0929 03:06:07.258352 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.258359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.258421 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:06:07.258652 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.258658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.258766 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.258873 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:07.259308 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.260215 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.261532 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.261923 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.262014 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.262980 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:07.262992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:07.263006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:07.263610 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:07.263658 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:07.263678 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:07.263767 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.263905 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.263991 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:07.264901 21983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:07.313512 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.312891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36448)
0929 03:06:07.313225 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.313331 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 03:06:07.314708 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.314318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36462)
0929 03:06:07.314485 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.314567 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:06:07.354445 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21861
I20260929 03:06:07.368397 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21991
I20260929 03:06:07.375137 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.369166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.2.205.131:34871)
0929 03:06:07.369357 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:07.369363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:07.369385 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:06:07.369544 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:07.369551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:07.369663 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:07.369760 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:07.370199 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.370930 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.372210 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:07.372607 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:07.372714 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:07.373870 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:07.373878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:07.373893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:07.374448 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:07.374495 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:07.374516 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:07.374633 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:07.374760 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:06:07.374864 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:06:07.375150 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.374686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36468)
0929 03:06:07.374867 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.374955 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 03:06:07.375725 22113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:07.376175 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.375787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36480)
0929 03:06:07.375938 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.376049 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:06:07.456996 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.456479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36496)
0929 03:06:07.456705 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.456822 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 03:06:07.458025 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:07.457664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:41591 (local address 127.0.0.1:36498)
0929 03:06:07.457844 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:07.457920 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 03:06:07.463734 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21991
I20260929 03:06:07.476883 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 21640
I20260929 03:06:07.477974 21701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:07.589794 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 21640
2026-09-29T03:06:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:07 dist-test-slave-530n krb5kdc[22171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:07 dist-test-slave-530n krb5kdc[22171](info): set up 2 sockets
Sep 29 03:06:07 dist-test-slave-530n krb5kdc[22171](info): commencing operation
krb5kdc: starting...
W20260929 03:06:08.609181 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:08 dist-test-slave-530n krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:08.856272 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35013
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:09.186157 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:09.186473 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:09.186582 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:09.194017 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:09.194144 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:09.194196 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:09.194275 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:09.194334 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:09.204499 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:09.205929 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:09.207286 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:09.219830 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:09.219745 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:09.219726 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:09.220654 22184 server_base.cc:1034] running on GCE node
I20260929 03:06:09.221495 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:09.223122 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:09.224375 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790651169224316 us; error 56 us; skew 500 ppm
I20260929 03:06:09.224798 22184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:09 dist-test-slave-530n krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:09.233417 22184 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:09.235512 22184 webserver.cc:460] Webserver started at http://127.2.205.190:43735/ using document root <none> and password file <none>
I20260929 03:06:09.236522 22184 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:09.236704 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:09.237141 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:09.240844 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "2237bff1df994c8d99ededadb678db28"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "50e1e4769db8ccf7b29883e62a82ad54"
server_key_iv: "68690e1575d9d24b9f0f75caf8df4de4"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.241671 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "2237bff1df994c8d99ededadb678db28"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "50e1e4769db8ccf7b29883e62a82ad54"
server_key_iv: "68690e1575d9d24b9f0f75caf8df4de4"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.247299 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 03:06:09.251046 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:09.252545 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:09.252725 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "2237bff1df994c8d99ededadb678db28"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "50e1e4769db8ccf7b29883e62a82ad54"
server_key_iv: "68690e1575d9d24b9f0f75caf8df4de4"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.252961 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:09.270181 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:09.274089 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:09.274343 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:09.292176 22184 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35013
I20260929 03:06:09.292183 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35013 every 8 connection(s)
I20260929 03:06:09.293686 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:09.297693 22251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:09.299633 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22184
I20260929 03:06:09.299810 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:09.300355 2870 external_mini_cluster.cc:1428] Setting key 7acbce5cb792e6dd98b2a9cc00a8877e
I20260929 03:06:09.311794 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: Bootstrap starting.
Sep 29 03:06:09 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:09.315514 22251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:09.316511 22251 log.cc:824] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:09.319377 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: No bootstrap required, opened a new log
I20260929 03:06:09.323555 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:09.302344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.2.205.190:35013)
0929 03:06:09.303180 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:09.303193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:09.303239 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:06:09.304864 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:09.304891 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:09.305302 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:09.305623 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:09.307485 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.308897 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.310752 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.311459 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.311602 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:09.316185 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:09.316194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:09.316208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:09.316248 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:09.319173 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:09.320024 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:09.320031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:09.320037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:09.320218 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:09.320636 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:09.320640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:09.320642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:09.321241 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:09.321555 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:09.322039 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 03:06:09.322496 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":380,"threads_started":1}
I20260929 03:06:09.327441 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } }
I20260929 03:06:09.327806 22251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:09.327896 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2237bff1df994c8d99ededadb678db28, State: Initialized, Role: FOLLOWER
I20260929 03:06:09.328421 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } }
I20260929 03:06:09.328647 22251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:09.328787 22251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:09.328938 22251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:09.331173 22251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } }
I20260929 03:06:09.331687 22251 leader_election.cc:302] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2237bff1df994c8d99ededadb678db28; no voters:
I20260929 03:06:09.332494 22251 leader_election.cc:288] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:09.332647 22256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:09.333618 22256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [term 1 LEADER]: Becoming Leader. State: Replica: 2237bff1df994c8d99ededadb678db28, State: Running, Role: LEADER
I20260929 03:06:09.334121 22256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } }
I20260929 03:06:09.334596 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:09.336939 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2237bff1df994c8d99ededadb678db28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } } }
I20260929 03:06:09.337024 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2237bff1df994c8d99ededadb678db28. Latest consensus state: current_term: 1 leader_uuid: "2237bff1df994c8d99ededadb678db28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2237bff1df994c8d99ededadb678db28" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35013 } } }
I20260929 03:06:09.337694 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:09.337695 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:09.344203 22264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:09.349332 22264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:09.358156 22264 catalog_manager.cc:1347] Generated new cluster ID: d74462a602c3420882de6b57b1e92150
I20260929 03:06:09.358294 22264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:09.377234 22264 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:09.378604 22264 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:09.385610 22264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: Generated new TSK 0
I20260929 03:06:09.386862 22264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.205.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
Entry for principal kudu/127.2.205.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab.
I20260929 03:06:09.460706 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:09.796171 22277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:09.796473 22277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:09.796603 22277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:09.804015 22277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:09.804162 22277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:09.804311 22277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:06:09.814378 22277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.129.keytab
--principal=kudu/127.2.205.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:09.815960 22277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:09.817543 22277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:09.831543 22282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:09.831622 22285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:09.831564 22283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:09.832458 22277 server_base.cc:1034] running on GCE node
I20260929 03:06:09.833173 22277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:09.834170 22277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:09.835367 22277 hybrid_clock.cc:648] HybridClock initialized: now 1790651169835315 us; error 40 us; skew 500 ppm
I20260929 03:06:09.835750 22277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:09 dist-test-slave-530n krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:09.839192 22277 init.cc:377] Logged in from keytab as kudu/127.2.205.129@KRBTEST.COM (short username kudu)
I20260929 03:06:09.840741 22277 webserver.cc:460] Webserver started at http://127.2.205.129:42705/ using document root <none> and password file <none>
I20260929 03:06:09.841531 22277 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:09.841645 22277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:09.841922 22277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:09.844992 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "397e52de527142539d3e5288a7fc9102"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "5ae523a25f596be355c014380e0b5a1f"
server_key_iv: "be0f5f80a93e7579a2836d157caea114"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.845734 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "397e52de527142539d3e5288a7fc9102"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "5ae523a25f596be355c014380e0b5a1f"
server_key_iv: "be0f5f80a93e7579a2836d157caea114"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.850696 22277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:09.854008 22292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:09.855423 22277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:09.855567 22277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "397e52de527142539d3e5288a7fc9102"
format_stamp: "Formatted at 2026-09-29 03:06:09 on dist-test-slave-530n"
server_key: "5ae523a25f596be355c014380e0b5a1f"
server_key_iv: "be0f5f80a93e7579a2836d157caea114"
server_key_version: "encryptionkey@0"
I20260929 03:06:09.855785 22277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:09.874816 22277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:09.878897 22277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:09.879145 22277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:09.880043 22277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:09.881505 22277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:09.881579 22277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:09.881678 22277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:09.881721 22277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:09.912204 22277 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:33287
I20260929 03:06:09.912245 22404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:33287 every 8 connection(s)
I20260929 03:06:09.913514 22277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:06:09.918066 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22277
I20260929 03:06:09.918229 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:06:09.918648 2870 external_mini_cluster.cc:1428] Setting key 70cf0988757341c97fea3e1224217035
Sep 29 03:06:09 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.129@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:09.935330 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:09.916095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:51047 (local address 127.2.205.190:35013)
0929 03:06:09.916532 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:09.916537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:09.917491 (+ 954us) server_negotiation.cc:407] Connection header received
0929 03:06:09.919467 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:09.919473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:09.919558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:09.919674 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:09.921597 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.922616 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.924193 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.924728 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.924873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:09.928872 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:09.928878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:09.928883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:09.928923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:09.931865 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:09.932663 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:09.932668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:09.932672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:09.932858 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:09.933372 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:09.933376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:09.933379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:09.933745 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:09.933891 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:09.934701 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 03:06:09.934947 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":233,"threads_started":1}
I20260929 03:06:09.936403 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:09.916578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35013 (local address 127.2.205.129:51047)
0929 03:06:09.917315 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:09.917365 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:09.919152 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:09.919950 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:09.919961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:09.920708 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:09.921397 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:09.921418 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.922780 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.922784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:09.924043 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:09.924057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:09.924980 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:09.924986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:09.925199 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:09.925852 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:09.925888 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:09.928601 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:09.932028 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:09.932036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:09.932051 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:09.932539 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:09.933005 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:09.933011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:09.933015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:09.933249 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:09.933898 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:09.933904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:09.934295 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 03:06:09.934568 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 03:06:09.935064 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":303,"threads_started":1}
I20260929 03:06:09.938261 22405 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35013
I20260929 03:06:09.938627 22405 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:09.939610 22405 heartbeater.cc:511] Master 127.2.205.190:35013 requested a full tablet report, sending...
I20260929 03:06:09.942346 22216 ts_manager.cc:194] Registered new tserver with Master: 397e52de527142539d3e5288a7fc9102 (127.2.205.129:33287)
I20260929 03:06:09.943886 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.129@KRBTEST.COM'} at 127.2.205.129:51047
WARNING: no policy specified for kudu/127.2.205.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
Entry for principal kudu/127.2.205.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab.
I20260929 03:06:09.974203 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:10.298061 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:10.298341 22411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:10.298472 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:10.305719 22411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:10.305842 22411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:10.306012 22411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:06:10.315644 22411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.130.keytab
--principal=kudu/127.2.205.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:10.316942 22411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:10.318266 22411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:10.331781 22416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:10.331809 22417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:10.331745 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:10.332499 22411 server_base.cc:1034] running on GCE node
I20260929 03:06:10.333365 22411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:10.334399 22411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:10.335608 22411 hybrid_clock.cc:648] HybridClock initialized: now 1790651170335534 us; error 72 us; skew 500 ppm
I20260929 03:06:10.335954 22411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:10 dist-test-slave-530n krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:10.338918 22411 init.cc:377] Logged in from keytab as kudu/127.2.205.130@KRBTEST.COM (short username kudu)
I20260929 03:06:10.340399 22411 webserver.cc:460] Webserver started at http://127.2.205.130:42483/ using document root <none> and password file <none>
I20260929 03:06:10.341136 22411 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:10.341250 22411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:10.341552 22411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:10.344419 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "dc07f7fe74cd4a379fd8e9684db333b9"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f9ea1d165e0f4af5dc7ac1164493b3d6"
server_key_iv: "80db99adb1db50976804f052bec43a4a"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.345374 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "dc07f7fe74cd4a379fd8e9684db333b9"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f9ea1d165e0f4af5dc7ac1164493b3d6"
server_key_iv: "80db99adb1db50976804f052bec43a4a"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.350346 22411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:06:10.353706 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.355002 22411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:06:10.355171 22411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "dc07f7fe74cd4a379fd8e9684db333b9"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f9ea1d165e0f4af5dc7ac1164493b3d6"
server_key_iv: "80db99adb1db50976804f052bec43a4a"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.355342 22411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:10.376771 22411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:10.381428 22411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:10.381734 22411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:10.382810 22411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:10.384326 22411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:10.384414 22411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.384532 22411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:10.384588 22411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.415956 22411 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:46617
I20260929 03:06:10.416005 22538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:46617 every 8 connection(s)
I20260929 03:06:10.417340 22411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:06:10.418262 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22411
I20260929 03:06:10.418478 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:06:10.419034 2870 external_mini_cluster.cc:1428] Setting key d3c0373c742560dff650eb3c6eb999fc
Sep 29 03:06:10 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.130@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:10.441622 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.420380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:41053 (local address 127.2.205.190:35013)
0929 03:06:10.420588 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:10.420595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:10.422194 (+ 1599us) server_negotiation.cc:407] Connection header received
0929 03:06:10.424968 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:10.424973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:10.425058 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:10.425171 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:10.427411 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.428347 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.430797 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.431357 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.431486 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:10.435544 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:10.435550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:10.435553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:10.435592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:10.438334 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.439158 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.439162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.439164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.439297 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.439754 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.439758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.439760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.440042 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:10.440176 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:10.441027 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 03:06:10.441268 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:06:10.442770 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.421042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35013 (local address 127.2.205.130:41053)
0929 03:06:10.422016 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:10.422076 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:10.424547 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:10.425484 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:10.425496 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:10.426302 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:10.427182 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:10.427209 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.428551 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.428556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:10.430628 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:10.430648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.431641 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.431648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:10.431963 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:10.432705 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:10.432767 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:10.435269 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:10.438506 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:10.438517 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:10.438537 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:10.439011 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:10.439417 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:10.439422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:10.439426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:10.439629 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:10.440174 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:10.440184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:10.440592 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 03:06:10.440909 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 03:06:10.441530 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":373,"threads_started":1}
I20260929 03:06:10.444383 22539 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35013
I20260929 03:06:10.444689 22539 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.205.131@KRBTEST.COM; defaulting to no policy
I20260929 03:06:10.445410 22539 heartbeater.cc:511] Master 127.2.205.190:35013 requested a full tablet report, sending...
I20260929 03:06:10.446857 22216 ts_manager.cc:194] Registered new tserver with Master: dc07f7fe74cd4a379fd8e9684db333b9 (127.2.205.130:46617)
I20260929 03:06:10.447726 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.130@KRBTEST.COM'} at 127.2.205.130:41053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
Entry for principal kudu/127.2.205.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab.
I20260929 03:06:10.474184 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:10.797065 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:10.797346 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:10.797468 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:10.804953 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:10.805086 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:10.805239 22544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:06:10.815299 22544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.131.keytab
--principal=kudu/127.2.205.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:35013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:10.816633 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:10.818002 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:10.831197 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:10.831187 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:10.831442 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:10.832048 22544 server_base.cc:1034] running on GCE node
I20260929 03:06:10.832726 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:10.833761 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:10.834990 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790651170834933 us; error 48 us; skew 500 ppm
I20260929 03:06:10.835418 22544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:10 dist-test-slave-530n krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:10.838651 22544 init.cc:377] Logged in from keytab as kudu/127.2.205.131@KRBTEST.COM (short username kudu)
I20260929 03:06:10.840260 22544 webserver.cc:460] Webserver started at http://127.2.205.131:40055/ using document root <none> and password file <none>
I20260929 03:06:10.841058 22544 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:10.841183 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:10.841486 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:10.844525 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "02ba9df6039f401aa104de00f42bf9ab"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f42a939be4efa8c3dc023ad3c9b41d63"
server_key_iv: "5a74070dd8cb55006ffbe87f0bb0112d"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.845355 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "02ba9df6039f401aa104de00f42bf9ab"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f42a939be4efa8c3dc023ad3c9b41d63"
server_key_iv: "5a74070dd8cb55006ffbe87f0bb0112d"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.850445 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:10.853773 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.855149 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:10.855289 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "02ba9df6039f401aa104de00f42bf9ab"
format_stamp: "Formatted at 2026-09-29 03:06:10 on dist-test-slave-530n"
server_key: "f42a939be4efa8c3dc023ad3c9b41d63"
server_key_iv: "5a74070dd8cb55006ffbe87f0bb0112d"
server_key_version: "encryptionkey@0"
I20260929 03:06:10.855460 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:10.871537 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:10.875526 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:10.875777 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:10.876709 22544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:10.878019 22544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:10.878088 22544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.878199 22544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:10.878252 22544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:10.908835 22544 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:40667
I20260929 03:06:10.908887 22671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:40667 every 8 connection(s)
I20260929 03:06:10.910203 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:06:10.917250 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22544
I20260929 03:06:10.917465 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:06:10.917925 2870 external_mini_cluster.cc:1428] Setting key de00b9b1cec582e9f62810f9e39e3749
Sep 29 03:06:10 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.131@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:10.932578 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.912920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:45675 (local address 127.2.205.190:35013)
0929 03:06:10.913147 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:10.913153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:10.914465 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 03:06:10.917142 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:10.917149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:10.917266 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:10.917397 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:10.919440 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.920388 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.922221 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.922658 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.922772 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:10.926563 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:10.926569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:10.926572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:10.926630 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:10.929314 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.930152 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.930157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.930160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.930317 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.930748 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.930751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.930753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.931092 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:10.931211 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:10.932017 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 03:06:10.932223 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:06:10.933789 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.913374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:35013 (local address 127.2.205.131:45675)
0929 03:06:10.914263 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:10.914320 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:10.916669 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:10.917743 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:10.917756 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:10.918414 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:10.919257 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:10.919281 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.920583 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.920589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:10.922016 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:10.922031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.922881 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.922886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:10.923170 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:10.923860 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:10.923884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:10.926292 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:10.929472 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:10.929480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:10.929503 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:10.929968 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:10.930421 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:10.930425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:10.930427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:10.930650 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:10.931206 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:10.931212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:10.931601 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 03:06:10.931846 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 03:06:10.932370 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
I20260929 03:06:10.935326 22672 heartbeater.cc:347] Connected to a master server at 127.2.205.190:35013
I20260929 03:06:10.935652 22672 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:10.936421 22672 heartbeater.cc:511] Master 127.2.205.190:35013 requested a full tablet report, sending...
I20260929 03:06:10.938103 22216 ts_manager.cc:194] Registered new tserver with Master: 02ba9df6039f401aa104de00f42bf9ab (127.2.205.131:40667)
I20260929 03:06:10.938956 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.205.131@KRBTEST.COM'} at 127.2.205.131:45675
I20260929 03:06:10.946286 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:06:10.947321 22405 heartbeater.cc:503] Master 127.2.205.190:35013 was elected leader, sending a full tablet report...
I20260929 03:06:10.963524 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.950600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.2.205.190:35013)
0929 03:06:10.950815 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:10.950820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:10.951142 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:06:10.951241 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:10.951244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:10.951307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:10.951394 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:10.952887 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.953673 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.955050 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:10.955470 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:10.955570 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:10.958220 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:10.958243 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:10.958246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:10.958279 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:10.961023 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.961642 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.961646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.961648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.961779 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:10.962136 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:10.962140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:10.962142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:10.962488 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:10.962673 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:10.963049 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:06:10.963229 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:06:10.967187 22216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:06:10.973788 22216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:06:11.008769 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.987121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.2.205.130:46617)
0929 03:06:11.000100 (+ 12979us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.000109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.001071 (+ 962us) server_negotiation.cc:407] Connection header received
0929 03:06:11.001161 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.001167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.001344 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.001490 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:11.002614 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.004262 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.006582 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.007599 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.007729 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.007963 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.008246 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.008418 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12818,"thread_start_us":12720,"threads_started":1}
I20260929 03:06:11.008908 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.986638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:46617 (local address 127.0.0.1:36434)
0929 03:06:11.000849 (+ 14211us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:11.000869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:11.000969 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:11.001752 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:11.001756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:11.001781 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:11.002401 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.002416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.004435 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.004439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.006386 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.006400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.007813 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.007819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.008000 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.008061 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 03:06:11.008170 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:06:11.008365 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14076,"mutex_wait_us":49,"spinlock_wait_cycles":896,"thread_start_us":13617,"threads_started":1}
I20260929 03:06:11.009685 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.985615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:33287 (local address 127.0.0.1:36474)
0929 03:06:11.000509 (+ 14894us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:11.000533 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:11.000680 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:11.001503 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:11.001510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:11.001538 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:11.002582 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.002596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.004079 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.004085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.006477 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.006493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.008817 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.008824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.009031 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.009056 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:06:11.009121 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:06:11.009317 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14763,"mutex_wait_us":619,"thread_start_us":12491,"threads_started":1}
I20260929 03:06:11.009743 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.985762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.2.205.131:40667)
0929 03:06:11.001250 (+ 15488us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.001256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.001283 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:06:11.001359 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.001364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.001604 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.002044 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:11.002619 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.004261 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.006807 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.008184 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.008306 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.008546 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.009268 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.009439 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15346,"thread_start_us":15188,"threads_started":1}
I20260929 03:06:11.009743 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.986035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.2.205.129:33287)
0929 03:06:10.999421 (+ 13386us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:10.999432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.000840 (+ 1408us) server_negotiation.cc:407] Connection header received
0929 03:06:11.000960 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.000969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.001264 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.001469 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:11.002788 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.003936 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.006673 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.008183 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.008306 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.008546 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.009229 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.009405 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13190,"thread_start_us":13383,"threads_started":1}
I20260929 03:06:11.010356 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:10.985542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:40667 (local address 127.0.0.1:55968)
0929 03:06:10.998534 (+ 12992us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:10.998579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:10.998768 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:11.001800 (+ 3032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:11.001803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:11.001821 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:11.002420 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.002429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.004400 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.004404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.006636 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.006648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.008716 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.008726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.008942 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.009019 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 03:06:11.009110 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:06:11.010006 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12730,"mutex_wait_us":792,"thread_start_us":15036,"threads_started":1}
I20260929 03:06:11.013375 22607 tablet_service.cc:1467] Processing CreateTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 (DEFAULT_TABLE table=test-table [id=73d30b782ab645d2b07c957a55612c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:11.013428 22340 tablet_service.cc:1467] Processing CreateTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 (DEFAULT_TABLE table=test-table [id=73d30b782ab645d2b07c957a55612c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:11.013428 22474 tablet_service.cc:1467] Processing CreateTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 (DEFAULT_TABLE table=test-table [id=73d30b782ab645d2b07c957a55612c4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:11.014719 22607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736032d0d65c4cfc92529d46828ea6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:11.014778 22340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736032d0d65c4cfc92529d46828ea6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:11.014778 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736032d0d65c4cfc92529d46828ea6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:11.026774 22690 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Bootstrap starting.
I20260929 03:06:11.030434 22690 tablet_bootstrap.cc:654] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:11.030781 22692 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Bootstrap starting.
I20260929 03:06:11.031937 22690 log.cc:824] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:11.034257 22692 tablet_bootstrap.cc:654] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:11.035209 22690 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: No bootstrap required, opened a new log
I20260929 03:06:11.035485 22690 ts_tablet_manager.cc:1397] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 03:06:11.035459 22692 log.cc:824] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:11.036830 22691 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Bootstrap starting.
I20260929 03:06:11.038347 22692 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: No bootstrap required, opened a new log
I20260929 03:06:11.038604 22692 ts_tablet_manager.cc:1397] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 03:06:11.039795 22691 tablet_bootstrap.cc:654] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:11.040800 22691 log.cc:824] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:11.043668 22691 tablet_bootstrap.cc:492] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: No bootstrap required, opened a new log
I20260929 03:06:11.043977 22691 ts_tablet_manager.cc:1397] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260929 03:06:11.045979 22690 raft_consensus.cc:348] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.046480 22690 raft_consensus.cc:374] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:11.046576 22690 raft_consensus.cc:729] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ba9df6039f401aa104de00f42bf9ab, State: Initialized, Role: FOLLOWER
I20260929 03:06:11.047313 22690 consensus_queue.cc:260] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.049196 22690 ts_tablet_manager.cc:1428] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 03:06:11.049175 22692 raft_consensus.cc:348] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.049634 22692 raft_consensus.cc:374] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:11.049664 22672 heartbeater.cc:503] Master 127.2.205.190:35013 was elected leader, sending a full tablet report...
I20260929 03:06:11.049729 22692 raft_consensus.cc:729] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397e52de527142539d3e5288a7fc9102, State: Initialized, Role: FOLLOWER
I20260929 03:06:11.050374 22692 consensus_queue.cc:260] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.052246 22692 ts_tablet_manager.cc:1428] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 03:06:11.052613 22691 raft_consensus.cc:348] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.053059 22691 raft_consensus.cc:374] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:11.053150 22691 raft_consensus.cc:729] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc07f7fe74cd4a379fd8e9684db333b9, State: Initialized, Role: FOLLOWER
I20260929 03:06:11.053870 22691 consensus_queue.cc:260] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.055020 22539 heartbeater.cc:503] Master 127.2.205.190:35013 was elected leader, sending a full tablet report...
I20260929 03:06:11.055694 22691 ts_tablet_manager.cc:1428] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260929 03:06:11.100277 22698 raft_consensus.cc:482] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:06:11.100575 22698 raft_consensus.cc:504] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.102320 22698 leader_election.cc:288] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ba9df6039f401aa104de00f42bf9ab (127.2.205.131:40667), 397e52de527142539d3e5288a7fc9102 (127.2.205.129:33287)
I20260929 03:06:11.109361 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.102783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:44251 (local address 127.2.205.131:40667)
0929 03:06:11.103021 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.103028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.103726 (+ 698us) server_negotiation.cc:407] Connection header received
0929 03:06:11.103899 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.103905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.103977 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.104050 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:11.104697 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.105521 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.107451 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.108279 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.108448 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.108534 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.108907 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.109076 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:06:11.109380 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.102818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:57099 (local address 127.2.205.129:33287)
0929 03:06:11.103048 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.103054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.103792 (+ 738us) server_negotiation.cc:407] Connection header received
0929 03:06:11.103943 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.103947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.104018 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.104105 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:11.104807 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.105550 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.107465 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.108286 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.108448 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.108534 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.108996 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.109143 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:06:11.109628 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.102683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:40667 (local address 127.2.205.130:44251)
0929 03:06:11.103589 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:11.103605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:11.103742 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:11.104102 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:11.104106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:11.104131 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:11.104527 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.104540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.105662 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.105668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.107255 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.107268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.108557 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.108564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.108708 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.108772 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 03:06:11.108851 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:06:11.109076 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":82,"thread_start_us":564,"threads_started":1}
I20260929 03:06:11.109982 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.102842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:33287 (local address 127.2.205.130:57099)
0929 03:06:11.103650 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:11.103668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:11.103803 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:11.104150 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:11.104154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:11.104177 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:11.104601 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.104614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.105696 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.105700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.107283 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:11.107296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.108598 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.108603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:11.108782 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.108818 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:06:11.108913 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:06:11.109091 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":142,"thread_start_us":456,"threads_started":1}
I20260929 03:06:11.110668 22627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736032d0d65c4cfc92529d46828ea6d8" candidate_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ba9df6039f401aa104de00f42bf9ab" is_pre_election: true
I20260929 03:06:11.110961 22360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736032d0d65c4cfc92529d46828ea6d8" candidate_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397e52de527142539d3e5288a7fc9102" is_pre_election: true
I20260929 03:06:11.111193 22627 raft_consensus.cc:2454] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc07f7fe74cd4a379fd8e9684db333b9 in term 0.
I20260929 03:06:11.111533 22360 raft_consensus.cc:2454] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc07f7fe74cd4a379fd8e9684db333b9 in term 0.
I20260929 03:06:11.111965 22427 leader_election.cc:302] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ba9df6039f401aa104de00f42bf9ab, dc07f7fe74cd4a379fd8e9684db333b9; no voters:
I20260929 03:06:11.112432 22698 raft_consensus.cc:2781] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:06:11.112586 22698 raft_consensus.cc:482] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:06:11.112689 22698 raft_consensus.cc:3037] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:11.115087 22698 raft_consensus.cc:504] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.115768 22698 leader_election.cc:288] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [CANDIDATE]: Term 1 election: Requested vote from peers 02ba9df6039f401aa104de00f42bf9ab (127.2.205.131:40667), 397e52de527142539d3e5288a7fc9102 (127.2.205.129:33287)
I20260929 03:06:11.116421 22627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736032d0d65c4cfc92529d46828ea6d8" candidate_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ba9df6039f401aa104de00f42bf9ab"
I20260929 03:06:11.116429 22360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736032d0d65c4cfc92529d46828ea6d8" candidate_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397e52de527142539d3e5288a7fc9102"
I20260929 03:06:11.116650 22627 raft_consensus.cc:3037] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:11.116655 22360 raft_consensus.cc:3037] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:11.119022 22627 raft_consensus.cc:2454] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc07f7fe74cd4a379fd8e9684db333b9 in term 1.
I20260929 03:06:11.119009 22360 raft_consensus.cc:2454] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc07f7fe74cd4a379fd8e9684db333b9 in term 1.
I20260929 03:06:11.119580 22427 leader_election.cc:302] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02ba9df6039f401aa104de00f42bf9ab, 397e52de527142539d3e5288a7fc9102, dc07f7fe74cd4a379fd8e9684db333b9; no voters:
I20260929 03:06:11.119907 22698 raft_consensus.cc:2781] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:11.120363 22698 raft_consensus.cc:686] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 1 LEADER]: Becoming Leader. State: Replica: dc07f7fe74cd4a379fd8e9684db333b9, State: Running, Role: LEADER
I20260929 03:06:11.120882 22698 consensus_queue.cc:237] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } }
I20260929 03:06:11.125846 22215 catalog_manager.cc:5495] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc07f7fe74cd4a379fd8e9684db333b9 (127.2.205.130). New cstate: current_term: 1 leader_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc07f7fe74cd4a379fd8e9684db333b9" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 46617 } health_report { overall_health: HEALTHY } } }
W20260929 03:06:11.161882 22673 tablet.cc:2369] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:06:11.165897 22406 tablet.cc:2369] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:11.167476 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.161573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.2.205.130:46617)
0929 03:06:11.161816 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.161823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.161844 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:06:11.161958 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.161963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.162128 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.162289 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:11.162711 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.163477 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.164904 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.165334 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.165448 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.166020 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:11.166823 (+ 803us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:11.166946 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.167109 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.167192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 03:06:11.169502 22540 tablet.cc:2369] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:11.176575 22627 raft_consensus.cc:1261] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 1 FOLLOWER]: Refusing update from remote peer dc07f7fe74cd4a379fd8e9684db333b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:11.176573 22360 raft_consensus.cc:1261] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 1 FOLLOWER]: Refusing update from remote peer dc07f7fe74cd4a379fd8e9684db333b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:11.177575 22698 consensus_queue.cc:1035] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397e52de527142539d3e5288a7fc9102" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:11.177866 22701 consensus_queue.cc:1035] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ba9df6039f401aa104de00f42bf9ab" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:11.190608 22704 mvcc.cc:205] Tried to move back new op lower bound from 7334507197129027584 to 7334507196915068928. Current Snapshot: MvccSnapshot[applied={T|T < 7334507197129027584}]
I20260929 03:06:11.201068 22214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:11.201331 22214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:11.204854 22214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2237bff1df994c8d99ededadb678db28: Sending DeleteTablet for 3 replicas of tablet 736032d0d65c4cfc92529d46828ea6d8
I20260929 03:06:11.205963 22607 tablet_service.cc:1514] Processing DeleteTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:55968
I20260929 03:06:11.206072 22340 tablet_service.cc:1514] Processing DeleteTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36474
I20260929 03:06:11.206203 22474 tablet_service.cc:1514] Processing DeleteTablet for tablet 736032d0d65c4cfc92529d46828ea6d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:11 UTC) from {username='kudu', principal='kudu/127.2.205.190@KRBTEST.COM'} at 127.0.0.1:36434
I20260929 03:06:11.207938 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22277
I20260929 03:06:11.219743 22715 tablet_replica.cc:330] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: stopping tablet replica
I20260929 03:06:11.220345 22715 raft_consensus.cc:2229] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:11.220994 22715 raft_consensus.cc:2258] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:11.221215 22714 tablet_replica.cc:330] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: stopping tablet replica
I20260929 03:06:11.221688 22714 raft_consensus.cc:2229] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:11.221853 22716 tablet_replica.cc:330] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: stopping tablet replica
I20260929 03:06:11.222232 22714 raft_consensus.cc:2258] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:11.222306 22716 raft_consensus.cc:2229] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:06:11.222925 22716 raft_consensus.cc:2258] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:11.225234 22715 ts_tablet_manager.cc:1905] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:11.225248 22716 ts_tablet_manager.cc:1905] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:11.232652 22715 ts_tablet_manager.cc:1918] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:11.232654 22716 ts_tablet_manager.cc:1918] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:11.232882 22715 log.cc:1196] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/736032d0d65c4cfc92529d46828ea6d8
I20260929 03:06:11.233081 22716 log.cc:1196] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/736032d0d65c4cfc92529d46828ea6d8
I20260929 03:06:11.233431 22715 ts_tablet_manager.cc:1939] T 736032d0d65c4cfc92529d46828ea6d8 P 397e52de527142539d3e5288a7fc9102: Deleting consensus metadata
I20260929 03:06:11.233773 22716 ts_tablet_manager.cc:1939] T 736032d0d65c4cfc92529d46828ea6d8 P dc07f7fe74cd4a379fd8e9684db333b9: Deleting consensus metadata
I20260929 03:06:11.235575 22203 catalog_manager.cc:4841] TS 397e52de527142539d3e5288a7fc9102 (127.2.205.129:33287): tablet 736032d0d65c4cfc92529d46828ea6d8 (table test-table [id=73d30b782ab645d2b07c957a55612c4f]) successfully deleted
I20260929 03:06:11.239148 22201 catalog_manager.cc:4841] TS dc07f7fe74cd4a379fd8e9684db333b9 (127.2.205.130:46617): tablet 736032d0d65c4cfc92529d46828ea6d8 (table test-table [id=73d30b782ab645d2b07c957a55612c4f]) successfully deleted
I20260929 03:06:11.237834 22714 ts_tablet_manager.cc:1905] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 03:06:11 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.129@KRBTEST.COM
I20260929 03:06:11.246891 22714 ts_tablet_manager.cc:1918] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:11.247069 22714 log.cc:1196] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/736032d0d65c4cfc92529d46828ea6d8
I20260929 03:06:11.247639 22714 ts_tablet_manager.cc:1939] T 736032d0d65c4cfc92529d46828ea6d8 P 02ba9df6039f401aa104de00f42bf9ab: Deleting consensus metadata
I20260929 03:06:11.249570 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.209039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.2.205.129:33287)
0929 03:06:11.209386 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.209392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.225380 (+ 15988us) server_negotiation.cc:407] Connection header received
0929 03:06:11.225696 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.225703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.225805 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.225884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:11.227762 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.228957 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.237998 (+ 9041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.238677 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.238793 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.243677 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:11.243692 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:11.243708 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:11.243749 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:11.246528 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.247334 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.247342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.247344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.247485 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.247910 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.247914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.247916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.248302 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:11.248603 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.249001 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.249229 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260929 03:06:11.249934 22200 catalog_manager.cc:4841] TS 02ba9df6039f401aa104de00f42bf9ab (127.2.205.131:40667): tablet 736032d0d65c4cfc92529d46828ea6d8 (table test-table [id=73d30b782ab645d2b07c957a55612c4f]) successfully deleted
I20260929 03:06:11.250510 22400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:11.368780 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22277
I20260929 03:06:11.383469 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22411
Sep 29 03:06:11 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.130@KRBTEST.COM
I20260929 03:06:11.399400 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.384270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.2.205.130:46617)
0929 03:06:11.384439 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.384445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.384468 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:06:11.384672 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.384677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.384800 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.384896 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:11.386174 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.386953 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.388241 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.388702 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.388861 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.393199 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:11.393214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:11.393230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:11.393271 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:11.396306 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.397255 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.397263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.397265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.397423 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.397800 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.397804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.397806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.398190 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:11.398529 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.398822 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.399064 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:06:11.400110 22534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:11.513720 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22411
I20260929 03:06:11.529994 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22544
Sep 29 03:06:11 dist-test-slave-530n krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.131@KRBTEST.COM
I20260929 03:06:11.546103 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:11.530924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.2.205.131:40667)
0929 03:06:11.531108 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:11.531114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:11.531135 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:06:11.531343 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:11.531348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:11.531427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:11.531534 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:11.532895 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.533660 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.535152 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:11.535607 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:11.535729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:11.540369 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:11.540386 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:11.540403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:11.540444 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:11.543266 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.543957 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.543966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.543968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.544110 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:11.544521 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:11.544525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:11.544527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:11.544948 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:11.545223 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:11.545562 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 03:06:11.545785 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:06:11.546741 22667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:11.636166 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22544
I20260929 03:06:11.649842 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22184
I20260929 03:06:11.650914 22246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:11.764400 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22184
2026-09-29T03:06:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:11Z Disabled control of system clock
I20260929 03:06:11.833900 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:42395
--webserver_interface=127.2.205.190
--webserver_port=0
--builtin_ntp_servers=127.2.205.148:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:06:12.164687 22725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:12.164988 22725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:12.165079 22725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:12.172111 22725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:12.172219 22725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:12.172254 22725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:12.172309 22725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:12.172354 22725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:12.181478 22725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:42395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:12.182788 22725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:12.184625 22725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:12.201444 22730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:12.201511 22733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:12.201483 22731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:12.202150 22725 server_base.cc:1034] running on GCE node
I20260929 03:06:12.203197 22725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:12.204800 22725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:12.206048 22725 hybrid_clock.cc:648] HybridClock initialized: now 1790651172205985 us; error 59 us; skew 500 ppm
I20260929 03:06:12.206434 22725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:12.208953 22725 webserver.cc:460] Webserver started at http://127.2.205.190:45191/ using document root <none> and password file <none>
I20260929 03:06:12.209703 22725 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:12.209813 22725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:12.210151 22725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:12.213287 22725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "12396cf8e4dd4c4caba8883053fc293b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "d7fcf9be57176c6ddf28f979edbc8d95"
server_key_iv: "5f199d553b12c63b4f5d043746fa852a"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.214172 22725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "12396cf8e4dd4c4caba8883053fc293b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "d7fcf9be57176c6ddf28f979edbc8d95"
server_key_iv: "5f199d553b12c63b4f5d043746fa852a"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.219563 22725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:12.223351 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:12.224989 22725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:12.225196 22725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "12396cf8e4dd4c4caba8883053fc293b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "d7fcf9be57176c6ddf28f979edbc8d95"
server_key_iv: "5f199d553b12c63b4f5d043746fa852a"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.225425 22725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:12.243305 22725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:12.244225 22725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:12.244488 22725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:12.261622 22725 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:42395
I20260929 03:06:12.261642 22790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:42395 every 8 connection(s)
I20260929 03:06:12.262878 22725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:12.266613 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:12.267675 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22725
I20260929 03:06:12.267824 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:12.268261 2870 external_mini_cluster.cc:1428] Setting key fdd6d3947d3d4647f502d353c796a7bf
I20260929 03:06:12.280743 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: Bootstrap starting.
I20260929 03:06:12.283218 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:12.270070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.2.205.190:42395)
0929 03:06:12.271160 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:12.271174 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:12.271232 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:06:12.272972 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:12.273009 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:12.273613 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:12.274065 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:12.274585 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.276423 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.277788 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.278848 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.279034 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:12.280163 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:12.280174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:12.280193 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:12.280947 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:12.281051 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:12.281080 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:12.281192 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:12.281438 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 03:06:12.281811 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":324,"threads_started":1}
I20260929 03:06:12.285037 22791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:12.286159 22791 log.cc:824] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:12.289314 22791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: No bootstrap required, opened a new log
I20260929 03:06:12.297138 22791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } }
I20260929 03:06:12.297570 22791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:12.297677 22791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12396cf8e4dd4c4caba8883053fc293b, State: Initialized, Role: FOLLOWER
I20260929 03:06:12.298352 22791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } }
I20260929 03:06:12.298609 22791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:12.298754 22791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:12.298920 22791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:12.301283 22791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } }
I20260929 03:06:12.301841 22791 leader_election.cc:302] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12396cf8e4dd4c4caba8883053fc293b; no voters:
I20260929 03:06:12.302479 22791 leader_election.cc:288] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:12.302613 22796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:12.303354 22796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [term 1 LEADER]: Becoming Leader. State: Replica: 12396cf8e4dd4c4caba8883053fc293b, State: Running, Role: LEADER
I20260929 03:06:12.303966 22796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } }
I20260929 03:06:12.304919 22791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:12.306917 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12396cf8e4dd4c4caba8883053fc293b. Latest consensus state: current_term: 1 leader_uuid: "12396cf8e4dd4c4caba8883053fc293b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } } }
I20260929 03:06:12.307343 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:12.307940 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12396cf8e4dd4c4caba8883053fc293b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12396cf8e4dd4c4caba8883053fc293b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 42395 } } }
I20260929 03:06:12.308393 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:12.313717 22801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:12.319965 22801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:12.330502 22801 catalog_manager.cc:1347] Generated new cluster ID: 697f5576d1b7406cac549d0bce971572
I20260929 03:06:12.330677 22801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:12.355165 22801 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:12.356705 22801 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:12.364321 22801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: Generated new TSK 0
I20260929 03:06:12.365443 22801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:12.370891 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.129:0
--local_ip_for_outbound_sockets=127.2.205.129
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--builtin_ntp_servers=127.2.205.148:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:06:12.692068 22815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:12.692315 22815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:12.692397 22815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:12.699256 22815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:12.699349 22815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:12.699448 22815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.129
I20260929 03:06:12.708624 22815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.205.129
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.129
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:12.709784 22815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:12.711453 22815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:12.725589 22820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:12.725566 22821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:12.725956 22823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:12.726358 22815 server_base.cc:1034] running on GCE node
I20260929 03:06:12.726986 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:12.727957 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:12.729197 22815 hybrid_clock.cc:648] HybridClock initialized: now 1790651172729148 us; error 35 us; skew 500 ppm
I20260929 03:06:12.729508 22815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:12.731820 22815 webserver.cc:460] Webserver started at http://127.2.205.129:35787/ using document root <none> and password file <none>
I20260929 03:06:12.732501 22815 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:12.732599 22815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:12.732944 22815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:12.735692 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data/instance:
uuid: "ba05d11163ba46658f15afee2cd4788b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "9d7792720bac9afd02c127f8a5639c24"
server_key_iv: "33c07cdf6ccf8ccc51fd55b16a577e5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.736438 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance:
uuid: "ba05d11163ba46658f15afee2cd4788b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "9d7792720bac9afd02c127f8a5639c24"
server_key_iv: "33c07cdf6ccf8ccc51fd55b16a577e5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.741339 22815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 03:06:12.744684 22829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:12.746058 22815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:12.746212 22815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal
uuid: "ba05d11163ba46658f15afee2cd4788b"
format_stamp: "Formatted at 2026-09-29 03:06:12 on dist-test-slave-530n"
server_key: "9d7792720bac9afd02c127f8a5639c24"
server_key_iv: "33c07cdf6ccf8ccc51fd55b16a577e5f"
server_key_version: "encryptionkey@0"
I20260929 03:06:12.746417 22815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:12.760620 22815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:12.761401 22815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:12.761711 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:12.762601 22815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:12.764000 22815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:12.764101 22815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:12.764209 22815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:12.764276 22815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:12.793890 22815 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.129:32891
I20260929 03:06:12.793929 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.129:32891 every 8 connection(s)
I20260929 03:06:12.795111 22815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/data/info.pb
I20260929 03:06:12.804267 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22815
I20260929 03:06:12.804423 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal/instance
I20260929 03:06:12.804946 2870 external_mini_cluster.cc:1428] Setting key b75db8582186b0d728eb0dd28f49b60e
I20260929 03:06:12.807560 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.130:0
--local_ip_for_outbound_sockets=127.2.205.130
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--builtin_ntp_servers=127.2.205.148:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:06:12.807641 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:12.797664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.129:41287 (local address 127.2.205.190:42395)
0929 03:06:12.798109 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:12.798113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:12.799261 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 03:06:12.801038 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:12.801048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:12.801122 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:12.801235 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:12.802366 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.803142 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.804704 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.805371 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.805501 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:12.806837 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:12.806843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:12.806848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:12.806867 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:12.806912 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:12.806924 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:12.806999 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:12.807216 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:06:12.807309 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":213,"threads_started":1}
I20260929 03:06:12.808585 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:12.798239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42395 (local address 127.2.205.129:41287)
0929 03:06:12.799109 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:12.799152 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:12.800741 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:12.801465 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:12.801474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:12.801564 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:12.802169 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:12.802188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.803279 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.803283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:12.804567 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:12.804579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:12.805592 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:12.805599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:12.805792 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:12.806530 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:12.806534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:12.806573 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:12.806663 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:12.807013 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:12.807024 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:12.807040 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:06:12.807192 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 03:06:12.807568 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":258,"threads_started":1}
I20260929 03:06:12.810091 22942 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42395
I20260929 03:06:12.810361 22942 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:12.811057 22942 heartbeater.cc:511] Master 127.2.205.190:42395 requested a full tablet report, sending...
I20260929 03:06:12.813140 22756 ts_manager.cc:194] Registered new tserver with Master: ba05d11163ba46658f15afee2cd4788b (127.2.205.129:32891)
I20260929 03:06:12.815059 22756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.129:41287
W20260929 03:06:13.114786 22946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:13.115057 22946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:13.115180 22946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:13.122320 22946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:13.122417 22946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:13.122563 22946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.130
I20260929 03:06:13.131686 22946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.205.130
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.130
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:13.132954 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:13.134677 22946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:13.146922 22954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:13.146986 22951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:13.146960 22952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:13.147869 22946 server_base.cc:1034] running on GCE node
I20260929 03:06:13.148413 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:13.149351 22946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:13.150579 22946 hybrid_clock.cc:648] HybridClock initialized: now 1790651173150527 us; error 43 us; skew 500 ppm
I20260929 03:06:13.150909 22946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:13.153131 22946 webserver.cc:460] Webserver started at http://127.2.205.130:45495/ using document root <none> and password file <none>
I20260929 03:06:13.153790 22946 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:13.153889 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:13.154197 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:13.156939 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data/instance:
uuid: "b8ff488f148d400ca4390d8d716adbc0"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "2a7d5e26f77e100e8f539b4d6a40a958"
server_key_iv: "b9d318e267adb14db7126df1e4a4dddd"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.157680 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance:
uuid: "b8ff488f148d400ca4390d8d716adbc0"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "2a7d5e26f77e100e8f539b4d6a40a958"
server_key_iv: "b9d318e267adb14db7126df1e4a4dddd"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.162470 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:06:13.165804 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.167096 22946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:13.167236 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal
uuid: "b8ff488f148d400ca4390d8d716adbc0"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "2a7d5e26f77e100e8f539b4d6a40a958"
server_key_iv: "b9d318e267adb14db7126df1e4a4dddd"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.167443 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:13.178689 22946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:13.179523 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:13.179817 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:13.180809 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:13.182238 22946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:13.182330 22946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.182441 22946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:13.182518 22946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.213838 22946 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.130:39659
I20260929 03:06:13.213884 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.130:39659 every 8 connection(s)
I20260929 03:06:13.215163 22946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/data/info.pb
I20260929 03:06:13.220733 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 22946
I20260929 03:06:13.220924 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal/instance
I20260929 03:06:13.221410 2870 external_mini_cluster.cc:1428] Setting key 0057740cdd543a24a579b167406a8372
I20260929 03:06:13.224400 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.205.131:0
--local_ip_for_outbound_sockets=127.2.205.131
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--builtin_ntp_servers=127.2.205.148:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:06:13.228351 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.217702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.130:35895 (local address 127.2.205.190:42395)
0929 03:06:13.217864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.217869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.219231 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 03:06:13.221298 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.221305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.221404 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.221520 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:13.222762 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.223587 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.225747 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.226145 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.226236 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.227690 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:13.227696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:13.227701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:13.227724 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:13.227756 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:13.227769 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:13.227838 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.228039 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.228115 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":55}
I20260929 03:06:13.229657 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.218197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42395 (local address 127.2.205.130:35895)
0929 03:06:13.219077 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.219118 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.220936 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.221835 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.221846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.221967 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:13.222594 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.222618 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.223744 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.223749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.225617 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.225631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.226345 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.226352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.226618 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.227295 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:13.227298 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:13.227343 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:13.227436 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:13.227852 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:13.227861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:13.227868 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.228036 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.228389 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":263,"threads_started":1}
I20260929 03:06:13.236886 23073 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42395
I20260929 03:06:13.237224 23073 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:13.238093 23073 heartbeater.cc:511] Master 127.2.205.190:42395 requested a full tablet report, sending...
I20260929 03:06:13.239764 22756 ts_manager.cc:194] Registered new tserver with Master: b8ff488f148d400ca4390d8d716adbc0 (127.2.205.130:39659)
I20260929 03:06:13.240824 22756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.130:35895
W20260929 03:06:13.544361 23076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:13.544636 23076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:13.544787 23076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:13.552169 23076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:13.552289 23076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:13.552428 23076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.205.131
I20260929 03:06:13.561411 23076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.205.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.205.131
--webserver_port=0
--tserver_master_addrs=127.2.205.190:42395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.205.131
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:13.562634 23076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:13.564395 23076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:13.577301 23081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:13.577371 23084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:13.577330 23082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:13.578136 23076 server_base.cc:1034] running on GCE node
I20260929 03:06:13.578760 23076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:13.579711 23076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:13.580912 23076 hybrid_clock.cc:648] HybridClock initialized: now 1790651173580846 us; error 56 us; skew 500 ppm
I20260929 03:06:13.581245 23076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:06:13.583462 23076 webserver.cc:460] Webserver started at http://127.2.205.131:36165/ using document root <none> and password file <none>
I20260929 03:06:13.584151 23076 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:13.584250 23076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:13.584551 23076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:13.587371 23076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data/instance:
uuid: "758fb734a86449fb8da7266641c0bf93"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "81c5fca5196e4e154f1440f331ae72d7"
server_key_iv: "eef3851104762077eab4b35053e85fbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.588104 23076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance:
uuid: "758fb734a86449fb8da7266641c0bf93"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "81c5fca5196e4e154f1440f331ae72d7"
server_key_iv: "eef3851104762077eab4b35053e85fbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.592908 23076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 03:06:13.596050 23090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.597325 23076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:06:13.597466 23076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal
uuid: "758fb734a86449fb8da7266641c0bf93"
format_stamp: "Formatted at 2026-09-29 03:06:13 on dist-test-slave-530n"
server_key: "81c5fca5196e4e154f1440f331ae72d7"
server_key_iv: "eef3851104762077eab4b35053e85fbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:13.597645 23076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:13.615530 23076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:13.616315 23076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:13.616585 23076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:13.617545 23076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:06:13.618906 23076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:06:13.619000 23076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.619102 23076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:06:13.619168 23076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:13.649212 23076 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.131:44741
I20260929 03:06:13.649284 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.131:44741 every 8 connection(s)
I20260929 03:06:13.650465 23076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/data/info.pb
I20260929 03:06:13.652554 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23076
I20260929 03:06:13.652709 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal/instance
I20260929 03:06:13.653250 2870 external_mini_cluster.cc:1428] Setting key abefd68f3344643f653e6ad91b8458fd
I20260929 03:06:13.662497 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.653044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:44695 (local address 127.2.205.190:42395)
0929 03:06:13.653255 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.653261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.654816 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 03:06:13.656544 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.656549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.656629 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.656715 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:13.657831 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.658548 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.660023 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.660421 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.660531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.661747 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:13.661753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:13.661757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:13.661774 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:13.661804 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:13.661816 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:13.661887 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.662091 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.662191 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:06:13.663394 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.653638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:42395 (local address 127.2.205.131:44695)
0929 03:06:13.654636 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.654698 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.656259 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.656976 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.656984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.657073 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:13.657665 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.657684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.658696 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.658700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.659890 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.659901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.660627 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.660634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.660889 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.661456 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:06:13.661458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:13.661495 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:06:13.661568 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:13.661904 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:13.661915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:13.661922 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.662063 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.662426 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":301,"threads_started":1}
I20260929 03:06:13.664629 23203 heartbeater.cc:347] Connected to a master server at 127.2.205.190:42395
I20260929 03:06:13.664924 23203 heartbeater.cc:464] Registering TS with master...
I20260929 03:06:13.665668 23203 heartbeater.cc:511] Master 127.2.205.190:42395 requested a full tablet report, sending...
I20260929 03:06:13.666985 22756 ts_manager.cc:194] Registered new tserver with Master: 758fb734a86449fb8da7266641c0bf93 (127.2.205.131:44741)
I20260929 03:06:13.667881 22756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.205.131:44695
I20260929 03:06:13.667979 2870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:06:13.677956 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.672570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.2.205.190:42395)
0929 03:06:13.672796 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.672802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.673232 (+ 430us) server_negotiation.cc:407] Connection header received
0929 03:06:13.673357 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.673361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.673433 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.673506 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:13.673923 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.674587 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.675856 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.676241 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.676331 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.677291 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:13.677295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:13.677298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:13.677315 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:13.677344 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:13.677355 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:13.677415 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.677584 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.677674 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:06:13.681953 22756 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:06:13.688565 22756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:06:13.723626 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.699939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.2.205.130:39659)
0929 03:06:13.712267 (+ 12328us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.712279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.713440 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 03:06:13.713932 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.713942 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.714244 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.714420 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:13.715576 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.717235 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.720384 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.721807 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.721900 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.722194 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.722969 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.723180 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12081,"thread_start_us":12351,"threads_started":1}
I20260929 03:06:13.724074 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.700298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.131:44741 (local address 127.0.0.1:45824)
0929 03:06:13.713686 (+ 13388us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.713705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.713836 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.715011 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.715016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.715045 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:13.715515 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.715527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.718342 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.718347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.720721 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.720738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.723248 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.723254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.723423 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.723455 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.723548 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.723756 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13239,"thread_start_us":13076,"threads_started":1}
I20260929 03:06:13.725054 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.699610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:39659 (local address 127.0.0.1:60310)
0929 03:06:13.713212 (+ 13602us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.713266 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.713646 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.714824 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.714832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.714869 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:13.715331 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.715350 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.717442 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.717448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.720126 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.720147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.722383 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.722390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.722660 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.722716 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.724474 (+ 1758us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.724647 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13264,"mutex_wait_us":2,"thread_start_us":13088,"threads_started":1}
I20260929 03:06:13.725081 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.700718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.2.205.131:44741)
0929 03:06:13.713069 (+ 12351us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.713078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.714001 (+ 923us) server_negotiation.cc:407] Connection header received
0929 03:06:13.714110 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.714117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.714478 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.714721 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:13.715730 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.718005 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.720973 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.722651 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.722757 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.723072 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.724516 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.724704 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12169,"thread_start_us":12332,"threads_started":1}
I20260929 03:06:13.729460 23008 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d (DEFAULT_TABLE table=test-table [id=1cdd532ef9b84ed38142961f4c0d3b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:13.729471 23138 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d (DEFAULT_TABLE table=test-table [id=1cdd532ef9b84ed38142961f4c0d3b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:13.730877 23008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f8ff322c645e6923bea1b36677e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:13.731024 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f8ff322c645e6923bea1b36677e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:13.734406 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.714320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57578)
0929 03:06:13.725562 (+ 11242us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.725583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.725706 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.728097 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.728102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.728129 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:13.728282 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.728291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.730208 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.730213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.731950 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.731966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.733677 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.733683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.733874 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.733909 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.734002 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.734135 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11126,"thread_start_us":11587,"threads_started":1}
I20260929 03:06:13.734546 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.714070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.2.205.129:32891)
0929 03:06:13.727500 (+ 13430us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.727508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.727538 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:06:13.727641 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.727647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.727883 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.728087 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:13.728516 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.730028 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.732131 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.733505 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.733645 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.733864 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.734056 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.734214 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13263,"thread_start_us":13124,"threads_started":1}
I20260929 03:06:13.738343 22877 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d (DEFAULT_TABLE table=test-table [id=1cdd532ef9b84ed38142961f4c0d3b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:06:13.739751 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0f8ff322c645e6923bea1b36677e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:13.742676 23221 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Bootstrap starting.
I20260929 03:06:13.745816 23222 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Bootstrap starting.
I20260929 03:06:13.746722 23221 tablet_bootstrap.cc:654] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:13.747938 23221 log.cc:824] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:13.749022 23222 tablet_bootstrap.cc:654] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:13.750075 23222 log.cc:824] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:13.750967 23221 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: No bootstrap required, opened a new log
I20260929 03:06:13.751266 23221 ts_tablet_manager.cc:1397] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 03:06:13.752692 23222 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: No bootstrap required, opened a new log
I20260929 03:06:13.752892 23225 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Bootstrap starting.
I20260929 03:06:13.754055 23222 ts_tablet_manager.cc:1397] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 03:06:13.756428 23225 tablet_bootstrap.cc:654] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:13.757697 23225 log.cc:824] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:13.759244 23221 raft_consensus.cc:348] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.759603 23221 raft_consensus.cc:374] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:13.759690 23221 raft_consensus.cc:729] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758fb734a86449fb8da7266641c0bf93, State: Initialized, Role: FOLLOWER
I20260929 03:06:13.760267 23221 consensus_queue.cc:260] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.760830 23225 tablet_bootstrap.cc:492] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: No bootstrap required, opened a new log
I20260929 03:06:13.761092 23225 ts_tablet_manager.cc:1397] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260929 03:06:13.761340 23203 heartbeater.cc:503] Master 127.2.205.190:42395 was elected leader, sending a full tablet report...
I20260929 03:06:13.761792 23221 ts_tablet_manager.cc:1428] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Time spent starting tablet: real 0.010s user 0.001s sys 0.009s
I20260929 03:06:13.764225 23222 raft_consensus.cc:348] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.764658 23222 raft_consensus.cc:374] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:13.764768 23222 raft_consensus.cc:729] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ff488f148d400ca4390d8d716adbc0, State: Initialized, Role: FOLLOWER
I20260929 03:06:13.765463 23222 consensus_queue.cc:260] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.766465 23073 heartbeater.cc:503] Master 127.2.205.190:42395 was elected leader, sending a full tablet report...
I20260929 03:06:13.767145 23222 ts_tablet_manager.cc:1428] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260929 03:06:13.771031 23225 raft_consensus.cc:348] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.771412 23225 raft_consensus.cc:374] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:13.771523 23225 raft_consensus.cc:729] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba05d11163ba46658f15afee2cd4788b, State: Initialized, Role: FOLLOWER
I20260929 03:06:13.772035 23225 consensus_queue.cc:260] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.772953 22942 heartbeater.cc:503] Master 127.2.205.190:42395 was elected leader, sending a full tablet report...
I20260929 03:06:13.773501 23225 ts_tablet_manager.cc:1428] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
W20260929 03:06:13.797211 22943 tablet.cc:2369] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:13.874835 23227 raft_consensus.cc:482] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:06:13.875121 23227 raft_consensus.cc:504] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.876871 23227 leader_election.cc:288] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8ff488f148d400ca4390d8d716adbc0 (127.2.205.130:39659), ba05d11163ba46658f15afee2cd4788b (127.2.205.129:32891)
I20260929 03:06:13.883226 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.877326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:50617 (local address 127.2.205.130:39659)
0929 03:06:13.877508 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.877514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.877555 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:06:13.877693 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.877698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.877776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.877862 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:13.878552 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.879493 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.881355 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.882341 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.882456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.882507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.882855 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.882961 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:06:13.883278 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.877219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.130:39659 (local address 127.2.205.131:50617)
0929 03:06:13.877402 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.877423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.877556 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.878064 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.878069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.878101 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:13.878329 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.878340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.879633 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.879639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.881198 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.881210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.882499 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.882505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.882685 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.882721 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.882822 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.882974 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 03:06:13.883882 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.877957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.205.131:52747 (local address 127.2.205.129:32891)
0929 03:06:13.878191 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.878196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.878382 (+ 186us) server_negotiation.cc:407] Connection header received
0929 03:06:13.878526 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.878531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.878599 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.878664 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:06:13.879232 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.880077 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.881946 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.882922 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.883087 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.883154 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.883474 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.883644 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 03:06:13.884008 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.877607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.2.205.131:52747)
0929 03:06:13.878285 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:13.878299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:13.878401 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:13.878704 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:13.878708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:13.878734 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:06:13.879088 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.879100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.880218 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.880222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.881744 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:13.881759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.883105 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.883111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:13.883279 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.883310 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:06:13.883629 (+ 319us) client_negotiation.cc:240] Negotiation successful
0929 03:06:13.883746 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":336,"threads_started":1}
I20260929 03:06:13.884364 23028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f8ff322c645e6923bea1b36677e9d" candidate_uuid: "758fb734a86449fb8da7266641c0bf93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ff488f148d400ca4390d8d716adbc0" is_pre_election: true
I20260929 03:06:13.884929 23028 raft_consensus.cc:2454] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758fb734a86449fb8da7266641c0bf93 in term 0.
I20260929 03:06:13.885005 22897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f8ff322c645e6923bea1b36677e9d" candidate_uuid: "758fb734a86449fb8da7266641c0bf93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba05d11163ba46658f15afee2cd4788b" is_pre_election: true
I20260929 03:06:13.885540 22897 raft_consensus.cc:2454] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758fb734a86449fb8da7266641c0bf93 in term 0.
I20260929 03:06:13.885550 23091 leader_election.cc:302] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758fb734a86449fb8da7266641c0bf93, b8ff488f148d400ca4390d8d716adbc0; no voters:
I20260929 03:06:13.885890 23227 raft_consensus.cc:2781] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:06:13.886037 23227 raft_consensus.cc:482] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:06:13.886150 23227 raft_consensus.cc:3037] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:13.888245 23227 raft_consensus.cc:504] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.888818 23227 leader_election.cc:288] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [CANDIDATE]: Term 1 election: Requested vote from peers b8ff488f148d400ca4390d8d716adbc0 (127.2.205.130:39659), ba05d11163ba46658f15afee2cd4788b (127.2.205.129:32891)
I20260929 03:06:13.889221 23028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f8ff322c645e6923bea1b36677e9d" candidate_uuid: "758fb734a86449fb8da7266641c0bf93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ff488f148d400ca4390d8d716adbc0"
I20260929 03:06:13.889281 22897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0f8ff322c645e6923bea1b36677e9d" candidate_uuid: "758fb734a86449fb8da7266641c0bf93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba05d11163ba46658f15afee2cd4788b"
I20260929 03:06:13.889402 23028 raft_consensus.cc:3037] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:13.889456 22897 raft_consensus.cc:3037] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:13.891554 23028 raft_consensus.cc:2454] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758fb734a86449fb8da7266641c0bf93 in term 1.
I20260929 03:06:13.891577 22897 raft_consensus.cc:2454] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758fb734a86449fb8da7266641c0bf93 in term 1.
I20260929 03:06:13.891902 23091 leader_election.cc:302] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758fb734a86449fb8da7266641c0bf93, b8ff488f148d400ca4390d8d716adbc0; no voters:
I20260929 03:06:13.892149 23227 raft_consensus.cc:2781] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:13.892573 23227 raft_consensus.cc:686] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 1 LEADER]: Becoming Leader. State: Replica: 758fb734a86449fb8da7266641c0bf93, State: Running, Role: LEADER
I20260929 03:06:13.893102 23227 consensus_queue.cc:237] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } }
I20260929 03:06:13.897816 22756 catalog_manager.cc:5495] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758fb734a86449fb8da7266641c0bf93 (127.2.205.131). New cstate: current_term: 1 leader_uuid: "758fb734a86449fb8da7266641c0bf93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758fb734a86449fb8da7266641c0bf93" member_type: VOTER last_known_addr { host: "127.2.205.131" port: 44741 } health_report { overall_health: HEALTHY } } }
W20260929 03:06:13.901911 23204 tablet.cc:2369] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:13.915762 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.910873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.2.205.131:44741)
0929 03:06:13.911084 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.911089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.911106 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:06:13.911258 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.911261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.911336 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.911452 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:13.911726 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.912472 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.913819 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.914244 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.914330 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.914774 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:13.915302 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:13.915413 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.915475 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.915535 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:06:13.923739 22897 raft_consensus.cc:1261] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 1 FOLLOWER]: Refusing update from remote peer 758fb734a86449fb8da7266641c0bf93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:13.923739 23028 raft_consensus.cc:1261] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 1 FOLLOWER]: Refusing update from remote peer 758fb734a86449fb8da7266641c0bf93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:13.924678 23231 consensus_queue.cc:1035] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba05d11163ba46658f15afee2cd4788b" member_type: VOTER last_known_addr { host: "127.2.205.129" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:13.925092 23227 consensus_queue.cc:1035] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ff488f148d400ca4390d8d716adbc0" member_type: VOTER last_known_addr { host: "127.2.205.130" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:13.941331 23236 mvcc.cc:205] Tried to move back new op lower bound from 7334507208381095936 to 7334507208270172160. Current Snapshot: MvccSnapshot[applied={T|T < 7334507208381095936}]
I20260929 03:06:13.951817 22754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:13.952085 22754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:06:13.955847 22754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12396cf8e4dd4c4caba8883053fc293b: Sending DeleteTablet for 3 replicas of tablet 6b0f8ff322c645e6923bea1b36677e9d
I20260929 03:06:13.957103 23138 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:13 UTC) from {username='slave'} at 127.0.0.1:45824
I20260929 03:06:13.957088 22877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:13 UTC) from {username='slave'} at 127.0.0.1:57578
I20260929 03:06:13.957093 23008 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0f8ff322c645e6923bea1b36677e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:06:13 UTC) from {username='slave'} at 127.0.0.1:60310
I20260929 03:06:13.958851 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22815
I20260929 03:06:13.969532 23244 tablet_replica.cc:330] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: stopping tablet replica
I20260929 03:06:13.970010 23244 raft_consensus.cc:2229] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:13.970554 23244 raft_consensus.cc:2258] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:13.971149 23245 tablet_replica.cc:330] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: stopping tablet replica
I20260929 03:06:13.971592 23245 raft_consensus.cc:2229] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:06:13.972406 23245 raft_consensus.cc:2258] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:13.975068 23244 ts_tablet_manager.cc:1905] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:13.981338 23246 tablet_replica.cc:330] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: stopping tablet replica
I20260929 03:06:13.981794 23246 raft_consensus.cc:2229] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:06:13.982311 23246 raft_consensus.cc:2258] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:06:13.983956 23245 ts_tablet_manager.cc:1905] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:13.984556 23246 ts_tablet_manager.cc:1905] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:06:13.986094 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:13.960241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.2.205.129:32891)
0929 03:06:13.960432 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:13.960439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:13.974977 (+ 14538us) server_negotiation.cc:407] Connection header received
0929 03:06:13.975522 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:13.975527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:13.975644 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:13.975718 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:13.976383 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.977542 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.979323 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:13.979972 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:13.980068 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:13.981704 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:13.981715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:13.981731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:13.985173 (+ 3442us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:13.985237 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:13.985262 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:13.985378 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:13.985586 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:06:13.985708 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:06:13.986483 23244 ts_tablet_manager.cc:1918] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:13.986720 23244 log.cc:1196] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-0/wal/wals/6b0f8ff322c645e6923bea1b36677e9d
I20260929 03:06:13.987229 22937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:13.990818 23245 ts_tablet_manager.cc:1918] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:13.991007 23245 log.cc:1196] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-2/wal/wals/6b0f8ff322c645e6923bea1b36677e9d
I20260929 03:06:13.991534 23245 ts_tablet_manager.cc:1939] T 6b0f8ff322c645e6923bea1b36677e9d P 758fb734a86449fb8da7266641c0bf93: Deleting consensus metadata
I20260929 03:06:13.993441 23246 ts_tablet_manager.cc:1918] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:06:13.993458 22741 catalog_manager.cc:4841] TS 758fb734a86449fb8da7266641c0bf93 (127.2.205.131:44741): tablet 6b0f8ff322c645e6923bea1b36677e9d (table test-table [id=1cdd532ef9b84ed38142961f4c0d3b77]) successfully deleted
I20260929 03:06:13.993605 23246 log.cc:1196] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Deleting WAL directory at /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650859567927-2870-0/minicluster-data/ts-1/wal/wals/6b0f8ff322c645e6923bea1b36677e9d
I20260929 03:06:13.994122 23246 ts_tablet_manager.cc:1939] T 6b0f8ff322c645e6923bea1b36677e9d P b8ff488f148d400ca4390d8d716adbc0: Deleting consensus metadata
I20260929 03:06:13.996024 22740 catalog_manager.cc:4841] TS b8ff488f148d400ca4390d8d716adbc0 (127.2.205.130:39659): tablet 6b0f8ff322c645e6923bea1b36677e9d (table test-table [id=1cdd532ef9b84ed38142961f4c0d3b77]) successfully deleted
I20260929 03:06:14.081885 23244 ts_tablet_manager.cc:1939] T 6b0f8ff322c645e6923bea1b36677e9d P ba05d11163ba46658f15afee2cd4788b: Deleting consensus metadata
I20260929 03:06:14.082669 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22815
W20260929 03:06:14.104674 22740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.205.129:32891 (error 108)
I20260929 03:06:14.104846 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22946
I20260929 03:06:14.106731 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.106176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57594)
0929 03:06:14.106415 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.106533 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 03:06:14.107003 22740 catalog_manager.cc:4584] TS ba05d11163ba46658f15afee2cd4788b (127.2.205.129:32891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0f8ff322c645e6923bea1b36677e9d: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
I20260929 03:06:14.112866 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.106180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.2.205.130:39659)
0929 03:06:14.106376 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:14.106382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:14.106401 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:06:14.106464 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:14.106470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:14.106604 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:14.106712 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:14.107308 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:14.108034 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:14.109897 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:14.110302 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:14.110397 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:14.111520 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:14.111528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:14.111542 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:14.112133 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:14.112183 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:14.112203 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:14.112284 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:14.112430 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:06:14.112517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:06:14.113590 23068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:14.160965 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.160391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57604)
0929 03:06:14.160652 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.160790 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 03:06:14.162259 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.161801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57606)
0929 03:06:14.161972 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.162070 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:06:14.201050 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22946
I20260929 03:06:14.216423 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23076
I20260929 03:06:14.223232 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.222727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57622)
0929 03:06:14.222962 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.223054 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 03:06:14.223587 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.217182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.2.205.131:44741)
0929 03:06:14.217365 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:14.217371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:14.217455 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:06:14.217680 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:14.217684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:14.217777 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:14.217864 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:14.218293 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:14.219085 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:14.220333 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:14.220841 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:14.220955 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:14.222100 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:14.222109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:14.222123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:14.222760 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:14.222828 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:14.222855 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:14.222974 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:14.223155 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:06:14.223257 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:06:14.224463 23198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:14.224550 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.224113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57638)
0929 03:06:14.224289 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.224379 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 03:06:14.305819 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.305303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57654)
0929 03:06:14.305522 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.305631 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 03:06:14.306901 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.306502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57658)
0929 03:06:14.306710 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.306785 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 03:06:14.380100 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23076
I20260929 03:06:14.404611 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 22725
I20260929 03:06:14.405771 22786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:14.512784 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:14.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.129:32891 (local address 127.0.0.1:57660)
0929 03:06:14.512412 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:14.512536 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.129:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:06:14.513142 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 22725
2026-09-29T03:06:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2751 ms)
[----------] 6 tests from EncryptionPolicyTest (20929 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:14 dist-test-slave-530n krb5kdc[23255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:14 dist-test-slave-530n krb5kdc[23255](info): set up 2 sockets
Sep 29 03:06:14 dist-test-slave-530n krb5kdc[23255](info): commencing operation
krb5kdc: starting...
W20260929 03:06:15.541244 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:15 dist-test-slave-530n krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:15.773842 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:16.099941 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:16.100257 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:16.100390 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:16.107997 23268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:16.108172 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:16.108260 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:16.108340 23268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:16.108397 23268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:16.118876 23268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:16.120383 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:16.121938 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:16.134694 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:16.134760 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:16.134742 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:16.135936 23268 server_base.cc:1034] running on GCE node
I20260929 03:06:16.136847 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:16.138494 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:16.139734 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790651176139678 us; error 58 us; skew 500 ppm
I20260929 03:06:16.140095 23268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:16 dist-test-slave-530n krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:16.143978 23268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:16.145787 23268 webserver.cc:460] Webserver started at http://127.0.0.1:37525/ using document root <none> and password file <none>
I20260929 03:06:16.146548 23268 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:16.146677 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:16.146997 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:16.150255 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "47f9fe610370449997dde611f068985c"
format_stamp: "Formatted at 2026-09-29 03:06:16 on dist-test-slave-530n"
server_key: "c75ec14e888b7d574215fafcb41cd6f0"
server_key_iv: "c3e3a272e08e026209736e9ac34d16d4"
server_key_version: "encryptionkey@0"
I20260929 03:06:16.151073 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "47f9fe610370449997dde611f068985c"
format_stamp: "Formatted at 2026-09-29 03:06:16 on dist-test-slave-530n"
server_key: "c75ec14e888b7d574215fafcb41cd6f0"
server_key_iv: "c3e3a272e08e026209736e9ac34d16d4"
server_key_version: "encryptionkey@0"
I20260929 03:06:16.156574 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:06:16.160256 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:16.161723 23268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:16.161892 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "47f9fe610370449997dde611f068985c"
format_stamp: "Formatted at 2026-09-29 03:06:16 on dist-test-slave-530n"
server_key: "c75ec14e888b7d574215fafcb41cd6f0"
server_key_iv: "c3e3a272e08e026209736e9ac34d16d4"
server_key_version: "encryptionkey@0"
I20260929 03:06:16.162099 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:16.201184 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:16.205646 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:16.205914 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:16.223095 23268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33947
I20260929 03:06:16.223136 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33947 every 8 connection(s)
I20260929 03:06:16.224409 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:16.226751 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23268
I20260929 03:06:16.226954 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:16.227641 2870 external_mini_cluster.cc:1428] Setting key ed74eb64a2a1577d683fd0d69e36fcda
I20260929 03:06:16.228566 23335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:16.243232 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c: Bootstrap starting.
I20260929 03:06:16.246762 23335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:16.248128 23335 log.cc:824] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:06:16 dist-test-slave-530n krb5kdc[23255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:06:16.251077 23335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c: No bootstrap required, opened a new log
I20260929 03:06:16.257371 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:16.229704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58477 (local address 127.0.0.1:33947)
0929 03:06:16.230501 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:16.230515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:16.230559 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:06:16.232181 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:16.232203 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:16.232625 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:16.233058 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:16.234838 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:16.236295 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:16.238264 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:16.239105 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:16.239258 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:16.249711 (+ 10453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:16.249720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:16.249737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:16.249776 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:16.252834 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:16.253751 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:16.253759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:16.253767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:16.253951 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:16.254450 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:16.254454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:16.254457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:16.255019 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:16.255368 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:16.255884 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 03:06:16.256217 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":343,"threads_started":1}
I20260929 03:06:16.259629 23335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } }
I20260929 03:06:16.260072 23335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:16.260180 23335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f9fe610370449997dde611f068985c, State: Initialized, Role: FOLLOWER
I20260929 03:06:16.260866 23335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } }
I20260929 03:06:16.261108 23335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:16.261268 23335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:16.261454 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:16.261976 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:16.264600 23335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } }
I20260929 03:06:16.265262 23335 leader_election.cc:302] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f9fe610370449997dde611f068985c; no voters:
I20260929 03:06:16.265981 23335 leader_election.cc:288] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:16.266088 23341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:16.268399 23335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:16.268862 23341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [term 1 LEADER]: Becoming Leader. State: Replica: 47f9fe610370449997dde611f068985c, State: Running, Role: LEADER
I20260929 03:06:16.269393 23341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } }
I20260929 03:06:16.272049 23346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f9fe610370449997dde611f068985c. Latest consensus state: current_term: 1 leader_uuid: "47f9fe610370449997dde611f068985c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } } }
I20260929 03:06:16.272413 23346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:16.274420 23344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f9fe610370449997dde611f068985c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f9fe610370449997dde611f068985c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33947 } } }
I20260929 03:06:16.274672 23352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:16.274701 23344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:16.281172 23352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:16.286387 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:16.268064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49425 (local address 127.0.0.1:33947)
0929 03:06:16.268235 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:16.268242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:16.268680 (+ 438us) server_negotiation.cc:407] Connection header received
0929 03:06:16.268815 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:16.268820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:16.268910 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:16.269025 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:16.271183 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:16.272533 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:16.274527 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:16.275359 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:16.275479 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:16.278512 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:16.278523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:16.278528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:16.278579 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:16.282394 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:16.283594 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:16.283601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:16.283605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:16.283795 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:16.284409 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:16.284414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:16.284418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:16.284992 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:16.285287 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:16.285751 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 03:06:16.285958 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:06:16.293038 23352 catalog_manager.cc:1347] Generated new cluster ID: 9e5db997f683443eaf68d4fc6591d053
I20260929 03:06:16.293429 23352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:16.309149 23352 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:16.311091 23352 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:16.320940 23352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f9fe610370449997dde611f068985c: Generated new TSK 0
I20260929 03:06:16.322109 23352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:16.328591 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23268
I20260929 03:06:16.330008 23330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:16.432835 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23268
W20260929 03:06:16.447360 23253 connection.cc:535] client connection to 127.0.0.1:33947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33947): Connection reset by peer (error 104)
2026-09-29T03:06:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:16 dist-test-slave-530n krb5kdc[23367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:16 dist-test-slave-530n krb5kdc[23367](info): set up 2 sockets
Sep 29 03:06:16 dist-test-slave-530n krb5kdc[23367](info): commencing operation
krb5kdc: starting...
W20260929 03:06:17.449059 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:17 dist-test-slave-530n krb5kdc[23367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:17.690040 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:18.029050 23380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:18.029394 23380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:18.029525 23380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:18.037031 23380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:18.037178 23380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:18.037256 23380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:18.037338 23380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:18.037403 23380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:18.047408 23380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:18.048785 23380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:18.050092 23380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:18.061224 23385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:18.061264 23386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:18.061124 23388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:18.062194 23380 server_base.cc:1034] running on GCE node
I20260929 03:06:18.062949 23380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:18.064307 23380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:18.065503 23380 hybrid_clock.cc:648] HybridClock initialized: now 1790651178065447 us; error 43 us; skew 500 ppm
I20260929 03:06:18.065840 23380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:18 dist-test-slave-530n krb5kdc[23367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:18.068806 23380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:18.070174 23380 webserver.cc:460] Webserver started at http://127.0.0.1:33527/ using document root <none> and password file <none>
I20260929 03:06:18.070808 23380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:18.070919 23380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:18.071204 23380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:18.074218 23380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "9d710f7df1b64055b3893cd3e5574c76"
format_stamp: "Formatted at 2026-09-29 03:06:18 on dist-test-slave-530n"
server_key: "fb8644673012b871e86b3d9acca7e12b"
server_key_iv: "b79b8a6538f5c5ef4b9946782ba1f068"
server_key_version: "encryptionkey@0"
I20260929 03:06:18.074934 23380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "9d710f7df1b64055b3893cd3e5574c76"
format_stamp: "Formatted at 2026-09-29 03:06:18 on dist-test-slave-530n"
server_key: "fb8644673012b871e86b3d9acca7e12b"
server_key_iv: "b79b8a6538f5c5ef4b9946782ba1f068"
server_key_version: "encryptionkey@0"
I20260929 03:06:18.080235 23380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:06:18.083986 23395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:18.085580 23380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:18.085757 23380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "9d710f7df1b64055b3893cd3e5574c76"
format_stamp: "Formatted at 2026-09-29 03:06:18 on dist-test-slave-530n"
server_key: "fb8644673012b871e86b3d9acca7e12b"
server_key_iv: "b79b8a6538f5c5ef4b9946782ba1f068"
server_key_version: "encryptionkey@0"
I20260929 03:06:18.085960 23380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:18.107105 23380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:18.111692 23380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:18.111989 23380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:18.129990 23380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42419
I20260929 03:06:18.130020 23446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42419 every 8 connection(s)
I20260929 03:06:18.131428 23380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:18.134498 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23380
I20260929 03:06:18.134675 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:18.135102 2870 external_mini_cluster.cc:1428] Setting key d1ac6e4d1a38925bc24117b0e68dcb01
I20260929 03:06:18.135457 23447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:18.151141 23447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76: Bootstrap starting.
I20260929 03:06:18.151336 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:18.137081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37913 (local address 127.0.0.1:42419)
0929 03:06:18.137946 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:18.137963 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:18.138015 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:06:18.140487 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:18.140525 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:18.141214 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:18.141759 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:18.142366 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:18.144407 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:18.145946 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:18.147032 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:18.147261 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:18.148386 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:18.148398 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:18.148420 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:18.149223 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:18.149320 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:18.149357 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:18.149507 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:18.149801 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:06:18.149995 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":362,"threads_started":1}
I20260929 03:06:18.154459 23447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:18.155151 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:18.155534 23447 log.cc:824] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:18.158319 23447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76: No bootstrap required, opened a new log
I20260929 03:06:18.165876 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:18.159621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37881 (local address 127.0.0.1:42419)
0929 03:06:18.159793 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:18.159801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:18.160109 (+ 308us) server_negotiation.cc:407] Connection header received
0929 03:06:18.160257 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:18.160262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:18.160341 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:18.160442 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:18.160884 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:18.162005 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:18.163300 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:18.163965 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:18.164090 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:18.165116 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:18.165126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:18.165131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:18.165163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:18.165210 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:18.165230 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:18.165337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:18.165477 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:06:18.165533 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:06:18.167207 23447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:06:18.167598 23447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:18.167682 23447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d710f7df1b64055b3893cd3e5574c76, State: Initialized, Role: FOLLOWER
I20260929 03:06:18.168169 23447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:06:18.168357 23447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:18.168445 23447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:18.168561 23447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:18.170681 23447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:06:18.171180 23447 leader_election.cc:302] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d710f7df1b64055b3893cd3e5574c76; no voters:
I20260929 03:06:18.171751 23447 leader_election.cc:288] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:18.171828 23457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:18.172461 23457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [term 1 LEADER]: Becoming Leader. State: Replica: 9d710f7df1b64055b3893cd3e5574c76, State: Running, Role: LEADER
I20260929 03:06:18.172999 23457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:06:18.173682 23447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:18.175729 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d710f7df1b64055b3893cd3e5574c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260929 03:06:18.175745 23459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d710f7df1b64055b3893cd3e5574c76. Latest consensus state: current_term: 1 leader_uuid: "9d710f7df1b64055b3893cd3e5574c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d710f7df1b64055b3893cd3e5574c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260929 03:06:18.176188 23459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:18.177080 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:18.181270 23464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:18.186339 23464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:18.195904 23464 catalog_manager.cc:1347] Generated new cluster ID: a3ff4affa8dd4ac8b64ac62f89db9d36
I20260929 03:06:18.196043 23464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:18.213649 23464 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:18.215107 23464 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:18.222334 23464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d710f7df1b64055b3893cd3e5574c76: Generated new TSK 0
I20260929 03:06:18.223259 23464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:18.230301 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23380
W20260929 03:06:18.231608 23442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37913
W20260929 03:06:18.232383 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:18.232594 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23380
W20260929 03:06:18.246649 23365 connection.cc:535] client connection to 127.0.0.1:42419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42419): Connection reset by peer (error 104)
2026-09-29T03:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1799 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:18 dist-test-slave-530n krb5kdc[23478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:18 dist-test-slave-530n krb5kdc[23478](info): set up 2 sockets
Sep 29 03:06:18 dist-test-slave-530n krb5kdc[23478](info): commencing operation
krb5kdc: starting...
W20260929 03:06:19.234180 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:19 dist-test-slave-530n krb5kdc[23478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:19.475711 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:19.803579 23491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:19.803910 23491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:19.804039 23491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:19.811530 23491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:19.811661 23491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:19.811708 23491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:19.811791 23491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:19.811843 23491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:19.822093 23491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:19.823489 23491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:19.824827 23491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:19.836769 23497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:19.836793 23496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:19.836710 23499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:19.837514 23491 server_base.cc:1034] running on GCE node
I20260929 03:06:19.838325 23491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:19.839783 23491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:19.840999 23491 hybrid_clock.cc:648] HybridClock initialized: now 1790651179840951 us; error 43 us; skew 500 ppm
I20260929 03:06:19.841356 23491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:19 dist-test-slave-530n krb5kdc[23478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:19.848037 23491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:19.849635 23491 webserver.cc:460] Webserver started at http://127.0.0.1:33975/ using document root <none> and password file <none>
I20260929 03:06:19.850313 23491 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:19.850411 23491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:19.850663 23491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:19.853582 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "b3c1e94060ae4afaaecf9e363ce13afa"
format_stamp: "Formatted at 2026-09-29 03:06:19 on dist-test-slave-530n"
server_key: "5d1520bd533228952a9c046cbb52e740"
server_key_iv: "cd5f41a56c3825bff99311852ead763d"
server_key_version: "encryptionkey@0"
I20260929 03:06:19.854269 23491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "b3c1e94060ae4afaaecf9e363ce13afa"
format_stamp: "Formatted at 2026-09-29 03:06:19 on dist-test-slave-530n"
server_key: "5d1520bd533228952a9c046cbb52e740"
server_key_iv: "cd5f41a56c3825bff99311852ead763d"
server_key_version: "encryptionkey@0"
I20260929 03:06:19.859010 23491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 03:06:19.862139 23506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:19.863488 23491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:06:19.863647 23491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "b3c1e94060ae4afaaecf9e363ce13afa"
format_stamp: "Formatted at 2026-09-29 03:06:19 on dist-test-slave-530n"
server_key: "5d1520bd533228952a9c046cbb52e740"
server_key_iv: "cd5f41a56c3825bff99311852ead763d"
server_key_version: "encryptionkey@0"
I20260929 03:06:19.863811 23491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:19.869164 23491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:19.872812 23491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:19.873057 23491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:19.890431 23491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35289
I20260929 03:06:19.890487 23557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35289 every 8 connection(s)
I20260929 03:06:19.891860 23491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:19.895713 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:19.899955 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23491
I20260929 03:06:19.900161 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:19.900696 2870 external_mini_cluster.cc:1428] Setting key 773f0a97791802bf00b62e469178cd6a
I20260929 03:06:19.908972 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:19.902699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54241 (local address 127.0.0.1:35289)
0929 03:06:19.903535 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:19.903547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:19.903591 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:06:19.905308 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:19.905334 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:19.905765 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:19.906141 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:19.907260 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:19.907267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:19.907283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:19.907298 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:19.907362 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:19.907384 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:19.907553 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:19.907711 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:06:19.907907 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":386,"threads_started":1}
I20260929 03:06:19.909478 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa: Bootstrap starting.
I20260929 03:06:19.912739 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:19.912791 23558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:19.914127 23558 log.cc:824] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:19.917943 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa: No bootstrap required, opened a new log
I20260929 03:06:19.920397 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:19.917987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56465 (local address 127.0.0.1:35289)
0929 03:06:19.918193 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:19.918197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:19.918510 (+ 313us) server_negotiation.cc:407] Connection header received
0929 03:06:19.918704 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:19.918708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:19.918758 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:19.918839 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:19.919774 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:19.919778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:19.919781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:19.919796 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:19.919840 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:19.919859 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:19.919953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:19.920058 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:06:19.920118 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:06:19.926271 23558 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260929 03:06:19.926695 23558 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:19.926812 23558 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c1e94060ae4afaaecf9e363ce13afa, State: Initialized, Role: FOLLOWER
I20260929 03:06:19.927448 23558 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260929 03:06:19.927690 23558 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:19.927791 23558 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:19.927920 23558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:19.930169 23558 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260929 03:06:19.930678 23558 leader_election.cc:302] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c1e94060ae4afaaecf9e363ce13afa; no voters:
I20260929 03:06:19.931252 23558 leader_election.cc:288] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:19.931401 23568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:19.932500 23568 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [term 1 LEADER]: Becoming Leader. State: Replica: b3c1e94060ae4afaaecf9e363ce13afa, State: Running, Role: LEADER
I20260929 03:06:19.933068 23568 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260929 03:06:19.933463 23558 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:19.936177 23569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } } }
I20260929 03:06:19.936617 23569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:19.936486 23570 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c1e94060ae4afaaecf9e363ce13afa. Latest consensus state: current_term: 1 leader_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c1e94060ae4afaaecf9e363ce13afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } } }
I20260929 03:06:19.936810 23570 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:19.939456 23577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:19.944877 23577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:19.953433 23577 catalog_manager.cc:1347] Generated new cluster ID: db235fc550fe4901b21b5209be9466b1
I20260929 03:06:19.953553 23577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:19.964579 23577 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:19.965009 23577 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:19.973397 23577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c1e94060ae4afaaecf9e363ce13afa: Generated new TSK 0
I20260929 03:06:19.974128 23577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:19.984421 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23491
W20260929 03:06:19.985535 23553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54241
W20260929 03:06:19.986253 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:19.986389 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23491
2026-09-29T03:06:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:20 dist-test-slave-530n krb5kdc[23589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:20 dist-test-slave-530n krb5kdc[23589](info): set up 2 sockets
Sep 29 03:06:20 dist-test-slave-530n krb5kdc[23589](info): commencing operation
krb5kdc: starting...
W20260929 03:06:21.005293 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:21 dist-test-slave-530n krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:21.256378 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:21.587723 23602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:21.588055 23602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:21.588172 23602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:21.595662 23602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:21.595793 23602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:21.595847 23602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:21.595933 23602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:21.595986 23602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:21.606215 23602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:21.607573 23602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:21.609004 23602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:21.620949 23610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:21.620968 23608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:21.621127 23607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:21.621685 23602 server_base.cc:1034] running on GCE node
I20260929 03:06:21.622520 23602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:21.624130 23602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:21.625347 23602 hybrid_clock.cc:648] HybridClock initialized: now 1790651181625299 us; error 40 us; skew 500 ppm
I20260929 03:06:21.625736 23602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:21 dist-test-slave-530n krb5kdc[23589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:21.628931 23602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:21.630409 23602 webserver.cc:460] Webserver started at http://127.0.0.1:32795/ using document root <none> and password file <none>
I20260929 03:06:21.631104 23602 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:21.631206 23602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:21.631469 23602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:21.634553 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "742d58df01434f87a8f506d11f865ef8"
format_stamp: "Formatted at 2026-09-29 03:06:21 on dist-test-slave-530n"
server_key: "ab914767386032451d0ce2ebe89e6137"
server_key_iv: "9add127c028973057e2b2d1a63ec7cbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:21.635311 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "742d58df01434f87a8f506d11f865ef8"
format_stamp: "Formatted at 2026-09-29 03:06:21 on dist-test-slave-530n"
server_key: "ab914767386032451d0ce2ebe89e6137"
server_key_iv: "9add127c028973057e2b2d1a63ec7cbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:21.641177 23602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 03:06:21.645087 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:21.646539 23602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:21.646723 23602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "742d58df01434f87a8f506d11f865ef8"
format_stamp: "Formatted at 2026-09-29 03:06:21 on dist-test-slave-530n"
server_key: "ab914767386032451d0ce2ebe89e6137"
server_key_iv: "9add127c028973057e2b2d1a63ec7cbc"
server_key_version: "encryptionkey@0"
I20260929 03:06:21.646919 23602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:21.669575 23602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:21.673331 23602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:21.673573 23602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:21.691242 23602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39967
I20260929 03:06:21.691244 23668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39967 every 8 connection(s)
I20260929 03:06:21.692732 23602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:21.696702 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:21.700146 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23602
I20260929 03:06:21.700325 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:21.700786 2870 external_mini_cluster.cc:1428] Setting key 81bb6d4d124a186f3726c8c1c2b44b1d
I20260929 03:06:21.709165 23669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8: Bootstrap starting.
I20260929 03:06:21.712648 23669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:21.713843 23669 log.cc:824] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:06:21 dist-test-slave-530n krb5kdc[23589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:06:21.717338 23669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8: No bootstrap required, opened a new log
I20260929 03:06:21.724375 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:21.702830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60635 (local address 127.0.0.1:39967)
0929 03:06:21.703647 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:21.703660 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:21.703705 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:06:21.705532 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:21.705562 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:21.706067 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:21.706488 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:21.707982 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:21.709434 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:21.710758 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:21.711505 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:21.711693 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:21.716365 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:21.716375 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:21.716393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:21.716432 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:21.719691 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:21.720680 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:21.720689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:21.720697 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:21.720929 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:21.721361 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:21.721365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:21.721367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:21.721958 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:21.722204 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:21.722728 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 03:06:21.723172 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":342,"threads_started":1}
I20260929 03:06:21.726348 23669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } }
I20260929 03:06:21.726765 23669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:21.726884 23669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742d58df01434f87a8f506d11f865ef8, State: Initialized, Role: FOLLOWER
I20260929 03:06:21.727564 23669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } }
I20260929 03:06:21.727813 23669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:21.727942 23669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:21.728102 23669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:21.728307 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:21.730401 23669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } }
I20260929 03:06:21.730903 23669 leader_election.cc:302] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742d58df01434f87a8f506d11f865ef8; no voters:
I20260929 03:06:21.731521 23669 leader_election.cc:288] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:21.731612 23677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:21.733381 23677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [term 1 LEADER]: Becoming Leader. State: Replica: 742d58df01434f87a8f506d11f865ef8, State: Running, Role: LEADER
I20260929 03:06:21.733942 23669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:21.734036 23677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } }
I20260929 03:06:21.745716 23679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742d58df01434f87a8f506d11f865ef8. Latest consensus state: current_term: 1 leader_uuid: "742d58df01434f87a8f506d11f865ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } } }
I20260929 03:06:21.746137 23679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:21.747215 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:21.732545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37067 (local address 127.0.0.1:39967)
0929 03:06:21.732723 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:21.732729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:21.733308 (+ 579us) server_negotiation.cc:407] Connection header received
0929 03:06:21.733588 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:21.733594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:21.733683 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:21.733745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:21.735423 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:21.736393 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:21.737839 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:21.738341 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:21.738463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:21.740953 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:21.740958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:21.740962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:21.740998 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:21.743840 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:21.744724 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:21.744729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:21.744732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:21.744919 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:21.745408 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:21.745413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:21.745416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:21.745876 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:21.746136 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:21.746744 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 03:06:21.746967 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:06:21.748816 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742d58df01434f87a8f506d11f865ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742d58df01434f87a8f506d11f865ef8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39967 } } }
I20260929 03:06:21.749086 23692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:21.749105 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:21.755065 23692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:21.764078 23692 catalog_manager.cc:1347] Generated new cluster ID: 147b45c9344c4bf9b3fb37d1fa2467fd
I20260929 03:06:21.764195 23692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:21.807101 23692 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:21.808498 23692 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:21.821739 23692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742d58df01434f87a8f506d11f865ef8: Generated new TSK 0
I20260929 03:06:21.822752 23692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:21.830436 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23602
I20260929 03:06:21.831590 23664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:21.927703 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23602
2026-09-29T03:06:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:22 dist-test-slave-530n krb5kdc[23702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:22 dist-test-slave-530n krb5kdc[23702](info): set up 2 sockets
Sep 29 03:06:22 dist-test-slave-530n krb5kdc[23702](info): commencing operation
krb5kdc: starting...
W20260929 03:06:22.933408 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:23 dist-test-slave-530n krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:23.177770 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:23.510319 23715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:23.510646 23715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:23.510790 23715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:23.518232 23715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:23.518378 23715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:23.518461 23715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:23.518548 23715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:23.518607 23715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:23.528954 23715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:23.530352 23715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:23.531760 23715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:23.544041 23723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:23.544018 23721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:23.544370 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:23.544970 23715 server_base.cc:1034] running on GCE node
I20260929 03:06:23.545768 23715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:23.547195 23715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:23.548432 23715 hybrid_clock.cc:648] HybridClock initialized: now 1790651183548374 us; error 44 us; skew 500 ppm
I20260929 03:06:23.548869 23715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:23 dist-test-slave-530n krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:23.551970 23715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:23.553442 23715 webserver.cc:460] Webserver started at http://127.0.0.1:33689/ using document root <none> and password file <none>
I20260929 03:06:23.554096 23715 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:23.554196 23715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:23.554443 23715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:23.557415 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "bff307df82ba485e90dba66ace55d795"
format_stamp: "Formatted at 2026-09-29 03:06:23 on dist-test-slave-530n"
server_key: "a2fbefad7201e95a7ba416fde7dce3a6"
server_key_iv: "cb199ccd7aa9027f9234fcf118c1dfc9"
server_key_version: "encryptionkey@0"
I20260929 03:06:23.558115 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "bff307df82ba485e90dba66ace55d795"
format_stamp: "Formatted at 2026-09-29 03:06:23 on dist-test-slave-530n"
server_key: "a2fbefad7201e95a7ba416fde7dce3a6"
server_key_iv: "cb199ccd7aa9027f9234fcf118c1dfc9"
server_key_version: "encryptionkey@0"
I20260929 03:06:23.563313 23715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:23.566694 23730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:23.568037 23715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 03:06:23.568192 23715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "bff307df82ba485e90dba66ace55d795"
format_stamp: "Formatted at 2026-09-29 03:06:23 on dist-test-slave-530n"
server_key: "a2fbefad7201e95a7ba416fde7dce3a6"
server_key_iv: "cb199ccd7aa9027f9234fcf118c1dfc9"
server_key_version: "encryptionkey@0"
I20260929 03:06:23.568363 23715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:23.587110 23715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:23.591765 23715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:23.592036 23715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:23.609370 23715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33085
I20260929 03:06:23.609427 23781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33085 every 8 connection(s)
I20260929 03:06:23.610827 23715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:23.612192 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23715
I20260929 03:06:23.612362 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:23.612926 2870 external_mini_cluster.cc:1428] Setting key 88d1c587582bc370518e3cd7cdf6c98c
I20260929 03:06:23.616123 23782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:23.626398 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:23.614624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34463 (local address 127.0.0.1:33085)
0929 03:06:23.615392 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:23.615406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:23.615457 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:06:23.617199 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:23.617225 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:23.617665 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:23.618035 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:23.618722 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:23.620130 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:23.621988 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:23.622703 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:23.622834 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:23.624081 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:23.624088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:23.624103 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:23.624655 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:23.624744 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:23.624799 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:23.624902 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:23.625088 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:06:23.625352 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":334,"threads_started":1}
I20260929 03:06:23.628908 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795: Bootstrap starting.
I20260929 03:06:23.630201 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:23.632400 23782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:23.633688 23782 log.cc:824] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:23.636736 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795: No bootstrap required, opened a new log
I20260929 03:06:23.641600 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:23.635126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37083 (local address 127.0.0.1:33085)
0929 03:06:23.635295 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:23.635301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:23.635861 (+ 560us) server_negotiation.cc:407] Connection header received
0929 03:06:23.636035 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:23.636039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:23.636121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:23.636197 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:23.636821 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:23.637565 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:23.639324 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:23.639801 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:23.639889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:23.640924 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:23.640928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:23.640931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:23.640948 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:23.640986 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:23.641001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:23.641068 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:23.641190 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:06:23.641265 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:06:23.645622 23782 raft_consensus.cc:348] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } }
I20260929 03:06:23.645999 23782 raft_consensus.cc:374] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:23.646103 23782 raft_consensus.cc:729] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff307df82ba485e90dba66ace55d795, State: Initialized, Role: FOLLOWER
I20260929 03:06:23.646662 23782 consensus_queue.cc:260] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } }
I20260929 03:06:23.646872 23782 raft_consensus.cc:388] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:23.647004 23782 raft_consensus.cc:482] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:23.647212 23782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:23.649557 23782 raft_consensus.cc:504] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } }
I20260929 03:06:23.650040 23782 leader_election.cc:302] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bff307df82ba485e90dba66ace55d795; no voters:
I20260929 03:06:23.650733 23782 leader_election.cc:288] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:23.650835 23792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:23.651473 23792 raft_consensus.cc:686] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [term 1 LEADER]: Becoming Leader. State: Replica: bff307df82ba485e90dba66ace55d795, State: Running, Role: LEADER
I20260929 03:06:23.652164 23792 consensus_queue.cc:237] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } }
I20260929 03:06:23.653162 23782 sys_catalog.cc:564] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:23.655435 23794 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bff307df82ba485e90dba66ace55d795. Latest consensus state: current_term: 1 leader_uuid: "bff307df82ba485e90dba66ace55d795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } } }
I20260929 03:06:23.655859 23794 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:23.655464 23793 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bff307df82ba485e90dba66ace55d795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff307df82ba485e90dba66ace55d795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33085 } } }
I20260929 03:06:23.656359 23793 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:23.658595 23800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:23.663692 23800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:23.673796 23800 catalog_manager.cc:1347] Generated new cluster ID: f443c89976f24da5a8ab38ccddc59272
I20260929 03:06:23.673904 23800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:23.684973 23800 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:23.687029 23800 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:23.699064 23800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bff307df82ba485e90dba66ace55d795: Generated new TSK 0
I20260929 03:06:23.700284 23800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:23.712353 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23715
W20260929 03:06:23.714007 23777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34463
W20260929 03:06:23.714890 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23715: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:23.715042 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23715
2026-09-29T03:06:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1786 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:23 dist-test-slave-530n krb5kdc[23813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:23 dist-test-slave-530n krb5kdc[23813](info): set up 2 sockets
Sep 29 03:06:23 dist-test-slave-530n krb5kdc[23813](info): commencing operation
krb5kdc: starting...
W20260929 03:06:24.729291 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:24 dist-test-slave-530n krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:24.971508 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:25.301656 23826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:25.301955 23826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:25.302074 23826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:25.309351 23826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:25.309458 23826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:25.309528 23826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:25.309603 23826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:25.309669 23826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:25.319710 23826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:25.321058 23826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:25.322311 23826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:25.333794 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:25.333837 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:25.333858 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:25.334610 23826 server_base.cc:1034] running on GCE node
I20260929 03:06:25.335336 23826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:25.336808 23826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:25.338034 23826 hybrid_clock.cc:648] HybridClock initialized: now 1790651185337986 us; error 56 us; skew 500 ppm
I20260929 03:06:25.338378 23826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:25 dist-test-slave-530n krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:25.344771 23826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:25.346263 23826 webserver.cc:460] Webserver started at http://127.0.0.1:42885/ using document root <none> and password file <none>
I20260929 03:06:25.346947 23826 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:25.347062 23826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:25.347355 23826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:25.350299 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "6afa47918a734682ad0ef208400ea28b"
format_stamp: "Formatted at 2026-09-29 03:06:25 on dist-test-slave-530n"
server_key: "66cb38f6088f755d4548c51640145c28"
server_key_iv: "f023302b5eea6e3a66af2281337c9458"
server_key_version: "encryptionkey@0"
I20260929 03:06:25.351024 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "6afa47918a734682ad0ef208400ea28b"
format_stamp: "Formatted at 2026-09-29 03:06:25 on dist-test-slave-530n"
server_key: "66cb38f6088f755d4548c51640145c28"
server_key_iv: "f023302b5eea6e3a66af2281337c9458"
server_key_version: "encryptionkey@0"
I20260929 03:06:25.356050 23826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 03:06:25.359431 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:25.360726 23826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:25.360924 23826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "6afa47918a734682ad0ef208400ea28b"
format_stamp: "Formatted at 2026-09-29 03:06:25 on dist-test-slave-530n"
server_key: "66cb38f6088f755d4548c51640145c28"
server_key_iv: "f023302b5eea6e3a66af2281337c9458"
server_key_version: "encryptionkey@0"
I20260929 03:06:25.361107 23826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:25.367185 23826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:25.371440 23826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:25.371719 23826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:25.389027 23826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40027
I20260929 03:06:25.389074 23892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40027 every 8 connection(s)
I20260929 03:06:25.390460 23826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:25.391145 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23826
I20260929 03:06:25.391302 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:25.391853 2870 external_mini_cluster.cc:1428] Setting key 4ce112dc22a55f776f62ef3c6a3e7602
I20260929 03:06:25.394795 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:25.401932 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:25.393821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38989 (local address 127.0.0.1:40027)
0929 03:06:25.394735 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:25.394753 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:25.395135 (+ 382us) server_negotiation.cc:407] Connection header received
0929 03:06:25.397550 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:25.397582 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:25.398100 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:25.398614 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:25.399662 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:25.399671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:25.399690 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:25.399711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:25.399796 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:25.399830 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:25.400072 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:25.400239 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:06:25.400476 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":369,"threads_started":1}
I20260929 03:06:25.406431 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:25.409718 23893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b: Bootstrap starting.
I20260929 03:06:25.413817 23893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:25.414388 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:25.411797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44017 (local address 127.0.0.1:40027)
0929 03:06:25.411978 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:25.411983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:25.412441 (+ 458us) server_negotiation.cc:407] Connection header received
0929 03:06:25.412613 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:25.412616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:25.412667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:25.412788 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:25.413738 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:25.413742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:25.413745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:25.413762 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:25.413809 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:25.413826 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:25.413928 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:25.414056 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:06:25.414107 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:06:25.415190 23893 log.cc:824] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:25.418009 23893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b: No bootstrap required, opened a new log
I20260929 03:06:25.425237 23893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } }
I20260929 03:06:25.425607 23893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:25.425689 23893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afa47918a734682ad0ef208400ea28b, State: Initialized, Role: FOLLOWER
I20260929 03:06:25.426221 23893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } }
I20260929 03:06:25.426424 23893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:25.426537 23893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:25.426672 23893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:25.429450 23893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } }
I20260929 03:06:25.429948 23893 leader_election.cc:302] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6afa47918a734682ad0ef208400ea28b; no voters:
I20260929 03:06:25.430510 23893 leader_election.cc:288] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:25.430677 23903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:25.431278 23903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [term 1 LEADER]: Becoming Leader. State: Replica: 6afa47918a734682ad0ef208400ea28b, State: Running, Role: LEADER
I20260929 03:06:25.431879 23903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } }
I20260929 03:06:25.432569 23893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:25.435220 23904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6afa47918a734682ad0ef208400ea28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } } }
I20260929 03:06:25.435678 23904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:25.437139 23905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6afa47918a734682ad0ef208400ea28b. Latest consensus state: current_term: 1 leader_uuid: "6afa47918a734682ad0ef208400ea28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afa47918a734682ad0ef208400ea28b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40027 } } }
I20260929 03:06:25.437593 23905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:25.438109 23909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:25.443949 23909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:25.454491 23909 catalog_manager.cc:1347] Generated new cluster ID: 329762e3743e40d4bcce4235fa57f7b1
I20260929 03:06:25.454609 23909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:25.472321 23909 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:25.472774 23909 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:25.481758 23909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6afa47918a734682ad0ef208400ea28b: Generated new TSK 0
I20260929 03:06:25.482679 23909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:25.499783 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23826
W20260929 03:06:25.501112 23888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38989
W20260929 03:06:25.501808 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:25.501938 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23826
2026-09-29T03:06:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1786 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:25 dist-test-slave-530n krb5kdc[23924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:25 dist-test-slave-530n krb5kdc[23924](info): set up 2 sockets
Sep 29 03:06:25 dist-test-slave-530n krb5kdc[23924](info): commencing operation
krb5kdc: starting...
W20260929 03:06:26.513281 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:26 dist-test-slave-530n krb5kdc[23924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:26.756176 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:27.083796 23937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:27.084129 23937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:27.084249 23937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:27.091821 23937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:27.091955 23937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:27.092005 23937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:27.092103 23937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:27.092160 23937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:27.102378 23937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:27.103839 23937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:27.105175 23937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:27.117197 23942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:27.117110 23943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:27.117156 23945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:27.118129 23937 server_base.cc:1034] running on GCE node
I20260929 03:06:27.119057 23937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:27.120841 23937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:27.122128 23937 hybrid_clock.cc:648] HybridClock initialized: now 1790651187122078 us; error 51 us; skew 500 ppm
I20260929 03:06:27.122575 23937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:27 dist-test-slave-530n krb5kdc[23924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:27.126224 23937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:27.127935 23937 webserver.cc:460] Webserver started at http://127.0.0.1:38965/ using document root <none> and password file <none>
I20260929 03:06:27.128675 23937 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:27.128837 23937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:27.129138 23937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:27.132205 23937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "fdffeb61a82247cf8eacb2b663be7b28"
format_stamp: "Formatted at 2026-09-29 03:06:27 on dist-test-slave-530n"
server_key: "e8ed388b592536c8de7167f34457bf5f"
server_key_iv: "4232b5c450c0d2d8e1c499dbc963f235"
server_key_version: "encryptionkey@0"
I20260929 03:06:27.133047 23937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "fdffeb61a82247cf8eacb2b663be7b28"
format_stamp: "Formatted at 2026-09-29 03:06:27 on dist-test-slave-530n"
server_key: "e8ed388b592536c8de7167f34457bf5f"
server_key_iv: "4232b5c450c0d2d8e1c499dbc963f235"
server_key_version: "encryptionkey@0"
I20260929 03:06:27.138417 23937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 03:06:27.142068 23952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:27.143504 23937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:27.143687 23937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "fdffeb61a82247cf8eacb2b663be7b28"
format_stamp: "Formatted at 2026-09-29 03:06:27 on dist-test-slave-530n"
server_key: "e8ed388b592536c8de7167f34457bf5f"
server_key_iv: "4232b5c450c0d2d8e1c499dbc963f235"
server_key_version: "encryptionkey@0"
I20260929 03:06:27.143879 23937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:27.156299 23937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:27.160506 23937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:27.160784 23937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:27.177906 23937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33349
I20260929 03:06:27.177928 24003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33349 every 8 connection(s)
I20260929 03:06:27.179517 23937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:27.183359 24004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:27.189029 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 23937
I20260929 03:06:27.189186 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:27.189607 2870 external_mini_cluster.cc:1428] Setting key c2c712a1730f1ce2f45b4dd96e7d9575
I20260929 03:06:27.194819 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28: Bootstrap starting.
I20260929 03:06:27.198417 24004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:27.199527 24004 log.cc:824] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:27.202324 24004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28: No bootstrap required, opened a new log
Sep 29 03:06:27 dist-test-slave-530n krb5kdc[23924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:06:27.211760 24004 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260929 03:06:27.212206 24004 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:27.212313 24004 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdffeb61a82247cf8eacb2b663be7b28, State: Initialized, Role: FOLLOWER
I20260929 03:06:27.213027 24004 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260929 03:06:27.213315 24004 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:27.213469 24004 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:27.213652 24004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:27.214241 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:27.191456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:43159 (local address 127.0.0.1:33349)
0929 03:06:27.192267 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:27.192284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:27.192340 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:06:27.194634 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:27.194661 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:27.195193 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:27.195611 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:27.196918 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:27.198832 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:27.200192 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:27.201025 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:27.201192 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:27.205237 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:27.205248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:27.205269 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:27.205312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:27.209200 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:27.209983 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:27.209992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:27.210000 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:27.210219 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:27.210691 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:27.210696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:27.210700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:27.211407 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:27.211694 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:27.212444 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 03:06:27.212827 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":318,"threads_started":1}
I20260929 03:06:27.216229 24004 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260929 03:06:27.216814 24004 leader_election.cc:302] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdffeb61a82247cf8eacb2b663be7b28; no voters:
I20260929 03:06:27.217437 24004 leader_election.cc:288] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:27.217548 24009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:27.218659 24009 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [term 1 LEADER]: Becoming Leader. State: Replica: fdffeb61a82247cf8eacb2b663be7b28, State: Running, Role: LEADER
I20260929 03:06:27.218751 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:27.219353 24009 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260929 03:06:27.219688 24004 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:27.228549 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdffeb61a82247cf8eacb2b663be7b28. Latest consensus state: current_term: 1 leader_uuid: "fdffeb61a82247cf8eacb2b663be7b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } } }
I20260929 03:06:27.229014 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:27.229156 24010 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdffeb61a82247cf8eacb2b663be7b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdffeb61a82247cf8eacb2b663be7b28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } } }
I20260929 03:06:27.229517 24010 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:27.239444 24022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:27.245289 24022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:27.251072 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:27.224915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:42281 (local address 127.0.0.1:33349)
0929 03:06:27.225138 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:27.225145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:27.231793 (+ 6648us) server_negotiation.cc:407] Connection header received
0929 03:06:27.232320 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:27.232327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:27.232420 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:27.232493 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:27.236772 (+ 4279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:27.237550 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:27.238949 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:27.239469 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:27.239602 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:27.242742 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:27.242749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:27.242754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:27.242794 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:27.246903 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:27.248075 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:27.248082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:27.248085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:27.248332 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:27.249082 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:27.249087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:27.249090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:27.249519 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:27.249786 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:27.250477 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 03:06:27.250688 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 03:06:27.259217 24022 catalog_manager.cc:1347] Generated new cluster ID: 7940c6d3a6334e59a7cdc6a641b26e15
I20260929 03:06:27.259411 24022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:27.284907 24022 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:27.286896 24022 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:27.297840 24022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdffeb61a82247cf8eacb2b663be7b28: Generated new TSK 0
I20260929 03:06:27.299033 24022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:27.312563 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 23937
I20260929 03:06:27.313977 23999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:27.410912 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 23937
W20260929 03:06:27.426810 23922 connection.cc:535] client connection to 127.0.0.1:33349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33349): Connection reset by peer (error 104)
2026-09-29T03:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:27 dist-test-slave-530n krb5kdc[24036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:27 dist-test-slave-530n krb5kdc[24036](info): set up 2 sockets
Sep 29 03:06:27 dist-test-slave-530n krb5kdc[24036](info): commencing operation
krb5kdc: starting...
W20260929 03:06:28.437304 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:28 dist-test-slave-530n krb5kdc[24036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:28.686215 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:29.014424 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:29.014768 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:29.014907 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:29.022326 24049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:29.022476 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:29.022562 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:29.022650 24049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:29.022712 24049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:29.032934 24049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:29.034375 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:29.035743 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:29.047554 24057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:29.047574 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:29.047601 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:29.048309 24049 server_base.cc:1034] running on GCE node
I20260929 03:06:29.049168 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:29.050643 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:29.051853 24049 hybrid_clock.cc:648] HybridClock initialized: now 1790651189051802 us; error 45 us; skew 500 ppm
I20260929 03:06:29.052186 24049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:29 dist-test-slave-530n krb5kdc[24036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:29.055240 24049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:29.056674 24049 webserver.cc:460] Webserver started at http://127.0.0.1:46581/ using document root <none> and password file <none>
I20260929 03:06:29.057354 24049 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:29.057467 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:29.057737 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:29.060628 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "fb39717c23c649549267bb0d3f7bb561"
format_stamp: "Formatted at 2026-09-29 03:06:29 on dist-test-slave-530n"
server_key: "2d5cba65b624ca282ba43bec28d3f84f"
server_key_iv: "20586d8690a0b548ad2403db0f32ec8c"
server_key_version: "encryptionkey@0"
I20260929 03:06:29.061378 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "fb39717c23c649549267bb0d3f7bb561"
format_stamp: "Formatted at 2026-09-29 03:06:29 on dist-test-slave-530n"
server_key: "2d5cba65b624ca282ba43bec28d3f84f"
server_key_iv: "20586d8690a0b548ad2403db0f32ec8c"
server_key_version: "encryptionkey@0"
I20260929 03:06:29.066169 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 03:06:29.069355 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:29.070567 24049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:29.070715 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "fb39717c23c649549267bb0d3f7bb561"
format_stamp: "Formatted at 2026-09-29 03:06:29 on dist-test-slave-530n"
server_key: "2d5cba65b624ca282ba43bec28d3f84f"
server_key_iv: "20586d8690a0b548ad2403db0f32ec8c"
server_key_version: "encryptionkey@0"
I20260929 03:06:29.070885 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:29.100975 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:29.105361 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:29.105666 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:29.123805 24049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35167
I20260929 03:06:29.123819 24115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35167 every 8 connection(s)
I20260929 03:06:29.125329 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:29.129347 24116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:29.130016 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24049
I20260929 03:06:29.130187 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:29.130743 2870 external_mini_cluster.cc:1428] Setting key 0776904f9c0ee002018e11c602f9d265
I20260929 03:06:29.144016 24116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561: Bootstrap starting.
I20260929 03:06:29.145922 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:29.132626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56339 (local address 127.0.0.1:35167)
0929 03:06:29.133473 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:29.133485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:29.133539 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:06:29.135890 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:29.135923 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:29.136431 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:29.136890 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:29.137471 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:29.138795 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:29.140892 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:29.141751 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:29.141899 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:29.143355 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:29.143364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:29.143382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:29.144010 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:29.144117 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:29.144155 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:29.144317 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:29.144607 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:06:29.144841 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":367,"threads_started":1}
I20260929 03:06:29.147771 24116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:29.148985 24116 log.cc:824] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:29.150152 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:29.152621 24116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561: No bootstrap required, opened a new log
I20260929 03:06:29.161610 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:29.155105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:49711 (local address 127.0.0.1:35167)
0929 03:06:29.155249 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:29.155254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:29.155580 (+ 326us) server_negotiation.cc:407] Connection header received
0929 03:06:29.155754 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:29.155761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:29.155824 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:29.155907 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:29.156445 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:29.157274 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:29.159095 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:29.159579 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:29.159670 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:29.160888 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:29.160892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:29.160895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:29.160911 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:29.160961 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:29.160977 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:29.161076 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:29.161249 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:06:29.161295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:06:29.162310 24116 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } }
I20260929 03:06:29.162703 24116 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:29.162801 24116 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb39717c23c649549267bb0d3f7bb561, State: Initialized, Role: FOLLOWER
I20260929 03:06:29.163422 24116 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } }
I20260929 03:06:29.163646 24116 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:29.163771 24116 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:29.163919 24116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:29.166085 24116 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } }
I20260929 03:06:29.166563 24116 leader_election.cc:302] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb39717c23c649549267bb0d3f7bb561; no voters:
I20260929 03:06:29.167168 24116 leader_election.cc:288] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:29.167274 24126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:29.167910 24126 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [term 1 LEADER]: Becoming Leader. State: Replica: fb39717c23c649549267bb0d3f7bb561, State: Running, Role: LEADER
I20260929 03:06:29.168416 24126 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } }
I20260929 03:06:29.169137 24116 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:29.171034 24127 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb39717c23c649549267bb0d3f7bb561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } } }
I20260929 03:06:29.171455 24127 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:29.171005 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb39717c23c649549267bb0d3f7bb561. Latest consensus state: current_term: 1 leader_uuid: "fb39717c23c649549267bb0d3f7bb561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39717c23c649549267bb0d3f7bb561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35167 } } }
I20260929 03:06:29.171869 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:29.173429 24131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:29.178867 24131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:29.188498 24131 catalog_manager.cc:1347] Generated new cluster ID: 4b2e93f3ef024baa90dee71c4e4d02f5
I20260929 03:06:29.188654 24131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:29.203231 24131 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:29.204996 24131 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:29.212116 24131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb39717c23c649549267bb0d3f7bb561: Generated new TSK 0
I20260929 03:06:29.213197 24131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:29.229319 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24049
W20260929 03:06:29.233385 24111 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:56339
W20260929 03:06:29.234405 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:29.234575 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24049
W20260929 03:06:29.250635 24034 connection.cc:535] client connection to 127.0.0.1:35167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35167): Connection reset by peer (error 104)
2026-09-29T03:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:29 dist-test-slave-530n krb5kdc[24147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:29 dist-test-slave-530n krb5kdc[24147](info): set up 2 sockets
Sep 29 03:06:29 dist-test-slave-530n krb5kdc[24147](info): commencing operation
krb5kdc: starting...
W20260929 03:06:30.249279 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:30 dist-test-slave-530n krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:30.492586 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:30.827965 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:30.828315 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:30.828452 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:30.836063 24160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:30.836197 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:30.836274 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:30.836352 24160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:30.836408 24160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:30.846801 24160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:30.848099 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:30.849449 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:30.860615 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:30.860651 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:30.860857 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:30.861301 24160 server_base.cc:1034] running on GCE node
I20260929 03:06:30.861980 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:30.863271 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:30.864466 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790651190864397 us; error 60 us; skew 500 ppm
I20260929 03:06:30.864799 24160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:30 dist-test-slave-530n krb5kdc[24147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:30.871119 24160 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:30.872507 24160 webserver.cc:460] Webserver started at http://127.0.0.1:34307/ using document root <none> and password file <none>
I20260929 03:06:30.873240 24160 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:30.873346 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:30.873634 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:30.876595 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "692946a2782546128b368c45c30fc13f"
format_stamp: "Formatted at 2026-09-29 03:06:30 on dist-test-slave-530n"
server_key: "a90fba9fc8365cda5601e097ed091aaf"
server_key_iv: "438c0068dff70833596991951731e5fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:30.877439 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "692946a2782546128b368c45c30fc13f"
format_stamp: "Formatted at 2026-09-29 03:06:30 on dist-test-slave-530n"
server_key: "a90fba9fc8365cda5601e097ed091aaf"
server_key_iv: "438c0068dff70833596991951731e5fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:30.882834 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:30.886698 24175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:30.888217 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:30.888412 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "692946a2782546128b368c45c30fc13f"
format_stamp: "Formatted at 2026-09-29 03:06:30 on dist-test-slave-530n"
server_key: "a90fba9fc8365cda5601e097ed091aaf"
server_key_iv: "438c0068dff70833596991951731e5fe"
server_key_version: "encryptionkey@0"
I20260929 03:06:30.888636 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:30.894672 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:30.898859 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:30.899196 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:30.917191 24160 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42551
I20260929 03:06:30.917229 24226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42551 every 8 connection(s)
I20260929 03:06:30.918604 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:30.922448 24227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:30.927000 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24160
I20260929 03:06:30.927193 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:30.927719 2870 external_mini_cluster.cc:1428] Setting key 832590b5e21c76f07c2bcabdc7233085
I20260929 03:06:30.934434 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f: Bootstrap starting.
I20260929 03:06:30.936412 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:30.929699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:52769 (local address 127.0.0.1:42551)
0929 03:06:30.930493 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:30.930505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:30.930547 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:06:30.932811 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:30.932839 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:30.933222 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:30.933575 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:30.934725 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:30.934732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:30.934747 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:30.934764 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:30.934849 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:30.934873 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:30.935099 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:30.935210 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 03:06:30.935423 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":344,"threads_started":1}
I20260929 03:06:30.937897 24227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:30.939016 24227 log.cc:824] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:30.940112 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:30.942349 24227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f: No bootstrap required, opened a new log
I20260929 03:06:30.948096 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:30.945558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:42793 (local address 127.0.0.1:42551)
0929 03:06:30.945755 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:30.945762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:30.946141 (+ 379us) server_negotiation.cc:407] Connection header received
0929 03:06:30.946291 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:30.946296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:30.946361 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:30.946479 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:30.947405 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:30.947415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:30.947420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:30.947438 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:30.947493 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:30.947514 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:30.947652 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:30.947704 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:06:30.947767 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:06:30.951375 24227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260929 03:06:30.951743 24227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:30.951838 24227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692946a2782546128b368c45c30fc13f, State: Initialized, Role: FOLLOWER
I20260929 03:06:30.952373 24227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260929 03:06:30.952591 24227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:30.952716 24227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:30.952906 24227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:30.955192 24227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260929 03:06:30.955668 24227 leader_election.cc:302] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692946a2782546128b368c45c30fc13f; no voters:
I20260929 03:06:30.956228 24227 leader_election.cc:288] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:30.956375 24237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:30.957306 24237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [term 1 LEADER]: Becoming Leader. State: Replica: 692946a2782546128b368c45c30fc13f, State: Running, Role: LEADER
I20260929 03:06:30.957876 24237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260929 03:06:30.958361 24227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:30.960551 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692946a2782546128b368c45c30fc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260929 03:06:30.960582 24239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692946a2782546128b368c45c30fc13f. Latest consensus state: current_term: 1 leader_uuid: "692946a2782546128b368c45c30fc13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692946a2782546128b368c45c30fc13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260929 03:06:30.961036 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:30.961033 24239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:30.962536 24244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:30.967933 24244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:30.974658 24254 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:06:30.979077 24244 catalog_manager.cc:1347] Generated new cluster ID: b22c98bddbe7409dac3efe0b9d525f1c
I20260929 03:06:30.979200 24244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:31.006860 24244 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:31.007476 24244 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:31.026259 24244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692946a2782546128b368c45c30fc13f: Generated new TSK 0
I20260929 03:06:31.027400 24244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:31.043939 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24160
W20260929 03:06:31.045218 24222 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:52769
W20260929 03:06:31.045971 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24160: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:31.046156 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24160
2026-09-29T03:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:31 dist-test-slave-530n krb5kdc[24258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:31 dist-test-slave-530n krb5kdc[24258](info): set up 2 sockets
Sep 29 03:06:31 dist-test-slave-530n krb5kdc[24258](info): commencing operation
krb5kdc: starting...
W20260929 03:06:32.073349 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:32 dist-test-slave-530n krb5kdc[24258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:32.324425 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:32.657686 24271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:32.658026 24271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:32.658171 24271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:32.665562 24271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:32.665689 24271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:32.665766 24271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:32.665844 24271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:32.665900 24271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:32.675810 24271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:32.677145 24271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:32.678484 24271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:32.690548 24277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:32.690608 24276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:32.690644 24279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:32.691331 24271 server_base.cc:1034] running on GCE node
I20260929 03:06:32.692199 24271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:32.693925 24271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:32.695158 24271 hybrid_clock.cc:648] HybridClock initialized: now 1790651192695103 us; error 39 us; skew 500 ppm
I20260929 03:06:32.695549 24271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:32 dist-test-slave-530n krb5kdc[24258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:32.698679 24271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:32.700328 24271 webserver.cc:460] Webserver started at http://127.0.0.1:45815/ using document root <none> and password file <none>
I20260929 03:06:32.701134 24271 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:32.701246 24271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:32.701537 24271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:32.704807 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "8edb08fbc73b422fa4b732f5d8e73d16"
format_stamp: "Formatted at 2026-09-29 03:06:32 on dist-test-slave-530n"
server_key: "8c0a7cb4e024071a4fdf48e1fee840c1"
server_key_iv: "bb9e588e81dad118ce39cdfad7a6c40b"
server_key_version: "encryptionkey@0"
I20260929 03:06:32.705667 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "8edb08fbc73b422fa4b732f5d8e73d16"
format_stamp: "Formatted at 2026-09-29 03:06:32 on dist-test-slave-530n"
server_key: "8c0a7cb4e024071a4fdf48e1fee840c1"
server_key_iv: "bb9e588e81dad118ce39cdfad7a6c40b"
server_key_version: "encryptionkey@0"
I20260929 03:06:32.711324 24271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:32.715227 24286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:32.716867 24271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:32.717046 24271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "8edb08fbc73b422fa4b732f5d8e73d16"
format_stamp: "Formatted at 2026-09-29 03:06:32 on dist-test-slave-530n"
server_key: "8c0a7cb4e024071a4fdf48e1fee840c1"
server_key_iv: "bb9e588e81dad118ce39cdfad7a6c40b"
server_key_version: "encryptionkey@0"
I20260929 03:06:32.717247 24271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:32.747225 24271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:32.751864 24271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:32.752141 24271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:32.769099 24271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40837
I20260929 03:06:32.769125 24337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40837 every 8 connection(s)
I20260929 03:06:32.770469 24271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:32.771803 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24271
I20260929 03:06:32.771968 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:32.772505 2870 external_mini_cluster.cc:1428] Setting key a620569eca0e2d3065f562cbd4c26aeb
I20260929 03:06:32.774951 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:06:32 dist-test-slave-530n krb5kdc[24258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:06:32.788177 24338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16: Bootstrap starting.
I20260929 03:06:32.792222 24338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:32.793434 24338 log.cc:824] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:32.796873 24338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16: No bootstrap required, opened a new log
I20260929 03:06:32.796927 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:32.774326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:41537 (local address 127.0.0.1:40837)
0929 03:06:32.775089 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:32.775101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:32.775143 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:06:32.776743 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:32.776801 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:32.777229 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:32.777589 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:32.779382 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:32.780851 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:32.782589 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:32.783278 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:32.783425 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:32.788201 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:32.788208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:32.788222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:32.788259 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:32.792035 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:32.792966 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:32.792974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:32.792982 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:32.793218 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:32.793670 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:32.793676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:32.793679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:32.794336 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:32.794583 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:32.795183 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 03:06:32.795547 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":313,"threads_started":1}
I20260929 03:06:32.801407 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:32.805588 24338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260929 03:06:32.806180 24338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:32.806283 24338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edb08fbc73b422fa4b732f5d8e73d16, State: Initialized, Role: FOLLOWER
I20260929 03:06:32.807054 24338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260929 03:06:32.807353 24338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:32.807502 24338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:32.807675 24338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:32.810724 24338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260929 03:06:32.811415 24338 leader_election.cc:302] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8edb08fbc73b422fa4b732f5d8e73d16; no voters:
I20260929 03:06:32.812237 24338 leader_election.cc:288] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:32.812391 24348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:32.813179 24348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [term 1 LEADER]: Becoming Leader. State: Replica: 8edb08fbc73b422fa4b732f5d8e73d16, State: Running, Role: LEADER
I20260929 03:06:32.813860 24348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260929 03:06:32.814538 24338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:32.816150 24349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8edb08fbc73b422fa4b732f5d8e73d16. Latest consensus state: current_term: 1 leader_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260929 03:06:32.816545 24349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:32.816949 24350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edb08fbc73b422fa4b732f5d8e73d16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260929 03:06:32.817258 24350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:32.824610 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:32.805913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:35733 (local address 127.0.0.1:40837)
0929 03:06:32.806116 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:32.806123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:32.806566 (+ 443us) server_negotiation.cc:407] Connection header received
0929 03:06:32.806778 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:32.806783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:32.806876 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:32.806987 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:32.808561 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:32.809793 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:32.811113 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:32.811751 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:32.811890 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:32.815203 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:32.815210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:32.815215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:32.815257 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:32.819401 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:32.820636 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:32.820643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:32.820647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:32.820912 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:32.821626 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:32.821632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:32.821636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:32.822050 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:32.823788 (+ 1738us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:32.823990 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:06:32.824205 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:06:32.828143 24353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:32.833685 24353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:32.844539 24353 catalog_manager.cc:1347] Generated new cluster ID: f3d41f9b76db4f9ea65ac71b85ff6c7f
I20260929 03:06:32.844679 24353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:32.861269 24353 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:32.863107 24353 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:32.875319 24353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8edb08fbc73b422fa4b732f5d8e73d16: Generated new TSK 0
I20260929 03:06:32.876195 24353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:32.884788 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24271
I20260929 03:06:32.885977 24333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:32.977828 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24271
W20260929 03:06:32.992481 24256 connection.cc:535] client connection to 127.0.0.1:40837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40837): Connection reset by peer (error 104)
2026-09-29T03:06:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:33 dist-test-slave-530n krb5kdc[24370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:33 dist-test-slave-530n krb5kdc[24370](info): set up 2 sockets
Sep 29 03:06:33 dist-test-slave-530n krb5kdc[24370](info): commencing operation
krb5kdc: starting...
W20260929 03:06:34.005616 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:34 dist-test-slave-530n krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:34.259565 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:34.591912 24383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:34.592269 24383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:34.592413 24383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:34.599840 24383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:34.600006 24383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:34.600097 24383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:34.600185 24383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:34.600248 24383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:34.610642 24383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:34.612082 24383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:34.613535 24383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:34.625694 24388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:34.625788 24391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:34.625883 24389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:34.626561 24383 server_base.cc:1034] running on GCE node
I20260929 03:06:34.627367 24383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:34.628901 24383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:34.630169 24383 hybrid_clock.cc:648] HybridClock initialized: now 1790651194630113 us; error 45 us; skew 500 ppm
I20260929 03:06:34.630560 24383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:34 dist-test-slave-530n krb5kdc[24370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:34.633704 24383 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:34.635123 24383 webserver.cc:460] Webserver started at http://127.0.0.1:42527/ using document root <none> and password file <none>
I20260929 03:06:34.635793 24383 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:34.635910 24383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:34.636193 24383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:34.639202 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "de78b8b1d3c1475bbda37a5764eac193"
format_stamp: "Formatted at 2026-09-29 03:06:34 on dist-test-slave-530n"
server_key: "229155c2a82b4057fab80e502723020a"
server_key_iv: "7fa3e5984d97f6101c6d5443ab80145b"
server_key_version: "encryptionkey@0"
I20260929 03:06:34.639919 24383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "de78b8b1d3c1475bbda37a5764eac193"
format_stamp: "Formatted at 2026-09-29 03:06:34 on dist-test-slave-530n"
server_key: "229155c2a82b4057fab80e502723020a"
server_key_iv: "7fa3e5984d97f6101c6d5443ab80145b"
server_key_version: "encryptionkey@0"
I20260929 03:06:34.645069 24383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:06:34.648327 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:34.649627 24383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:34.649777 24383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "de78b8b1d3c1475bbda37a5764eac193"
format_stamp: "Formatted at 2026-09-29 03:06:34 on dist-test-slave-530n"
server_key: "229155c2a82b4057fab80e502723020a"
server_key_iv: "7fa3e5984d97f6101c6d5443ab80145b"
server_key_version: "encryptionkey@0"
I20260929 03:06:34.649950 24383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:34.682224 24383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:34.686602 24383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:34.686919 24383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:34.705366 24383 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38451
I20260929 03:06:34.705389 24449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38451 every 8 connection(s)
I20260929 03:06:34.706933 24383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:34.711231 24450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:34.714224 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24383
I20260929 03:06:34.714414 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:34.714960 2870 external_mini_cluster.cc:1428] Setting key 08bb7fe882016a7dd092247a0d092820
I20260929 03:06:34.724459 24450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193: Bootstrap starting.
I20260929 03:06:34.728828 24450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:34.730089 24450 log.cc:824] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:34.730559 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:34.717145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:55625 (local address 127.0.0.1:38451)
0929 03:06:34.717960 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:34.717973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:34.718019 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:06:34.719814 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:34.719840 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:34.720343 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:34.720798 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:34.721499 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:34.722995 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:34.725121 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:34.725968 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:34.726192 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:34.727789 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:34.727799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:34.727817 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:34.728471 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:34.728573 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:34.728607 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:34.728827 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:34.729169 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 03:06:34.729404 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":350,"threads_started":1}
I20260929 03:06:34.733453 24450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193: No bootstrap required, opened a new log
I20260929 03:06:34.734848 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:34.742878 24450 raft_consensus.cc:348] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260929 03:06:34.743301 24450 raft_consensus.cc:374] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:34.743407 24450 raft_consensus.cc:729] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de78b8b1d3c1475bbda37a5764eac193, State: Initialized, Role: FOLLOWER
I20260929 03:06:34.744047 24450 consensus_queue.cc:260] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260929 03:06:34.744298 24450 raft_consensus.cc:388] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:34.744446 24450 raft_consensus.cc:482] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:34.744611 24450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:34.746201 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:34.740082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:44703 (local address 127.0.0.1:38451)
0929 03:06:34.740294 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:34.740301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:34.740623 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:06:34.740823 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:34.740828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:34.740901 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:34.740998 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:34.741511 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:34.742288 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:34.743692 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:34.744137 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:34.744247 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:34.745454 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:34.745459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:34.745463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:34.745482 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:34.745552 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:34.745569 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:34.745678 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:34.745874 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:06:34.745932 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:06:34.747263 24450 raft_consensus.cc:504] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260929 03:06:34.747774 24450 leader_election.cc:302] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de78b8b1d3c1475bbda37a5764eac193; no voters:
I20260929 03:06:34.748359 24450 leader_election.cc:288] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:34.748497 24460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:34.750708 24450 sys_catalog.cc:564] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:34.753093 24460 raft_consensus.cc:686] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [term 1 LEADER]: Becoming Leader. State: Replica: de78b8b1d3c1475bbda37a5764eac193, State: Running, Role: LEADER
I20260929 03:06:34.753865 24460 consensus_queue.cc:237] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260929 03:06:34.757225 24462 sys_catalog.cc:455] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de78b8b1d3c1475bbda37a5764eac193. Latest consensus state: current_term: 1 leader_uuid: "de78b8b1d3c1475bbda37a5764eac193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260929 03:06:34.757710 24462 sys_catalog.cc:458] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:34.757228 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de78b8b1d3c1475bbda37a5764eac193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de78b8b1d3c1475bbda37a5764eac193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260929 03:06:34.758095 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:34.758562 24473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:34.763933 24473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:34.773413 24473 catalog_manager.cc:1347] Generated new cluster ID: e3a5ad2ce1404604922c1467c3a90243
I20260929 03:06:34.773522 24473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:34.797791 24473 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:34.799631 24473 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:34.810683 24473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de78b8b1d3c1475bbda37a5764eac193: Generated new TSK 0
I20260929 03:06:34.811843 24473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:34.828497 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24383
W20260929 03:06:34.829792 24445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:55625
W20260929 03:06:34.830703 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:34.830869 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24383
W20260929 03:06:34.845502 24368 connection.cc:535] client connection to 127.0.0.1:38451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38451): Connection reset by peer (error 104)
2026-09-29T03:06:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:34 dist-test-slave-530n krb5kdc[24481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:34 dist-test-slave-530n krb5kdc[24481](info): set up 2 sockets
Sep 29 03:06:34 dist-test-slave-530n krb5kdc[24481](info): commencing operation
krb5kdc: starting...
W20260929 03:06:35.861311 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:36 dist-test-slave-530n krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:06:36.104957 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:36.437299 24494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:36.437655 24494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:36.437795 24494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:36.446236 24494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:36.446477 24494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:36.446607 24494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:36.446707 24494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:36.446768 24494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:36.461938 24494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:36.463541 24494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:36.465078 24494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:36.480901 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:36.480870 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:36.480937 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:36.482076 24494 server_base.cc:1034] running on GCE node
I20260929 03:06:36.482987 24494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:36.484640 24494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:36.485864 24494 hybrid_clock.cc:648] HybridClock initialized: now 1790651196485799 us; error 64 us; skew 500 ppm
I20260929 03:06:36.486270 24494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:36 dist-test-slave-530n krb5kdc[24481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:36.489661 24494 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:06:36.491575 24494 webserver.cc:460] Webserver started at http://127.0.0.1:33279/ using document root <none> and password file <none>
I20260929 03:06:36.492275 24494 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:36.492381 24494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:36.492635 24494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:36.495747 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "dcbe14ad4a8745719f2c2a71b37151ba"
format_stamp: "Formatted at 2026-09-29 03:06:36 on dist-test-slave-530n"
server_key: "6ef950f6bfe8e28e8dedad00e36637d4"
server_key_iv: "22d71e84bdef7344e4384d9c55c9f95d"
server_key_version: "encryptionkey@0"
I20260929 03:06:36.496481 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "dcbe14ad4a8745719f2c2a71b37151ba"
format_stamp: "Formatted at 2026-09-29 03:06:36 on dist-test-slave-530n"
server_key: "6ef950f6bfe8e28e8dedad00e36637d4"
server_key_iv: "22d71e84bdef7344e4384d9c55c9f95d"
server_key_version: "encryptionkey@0"
I20260929 03:06:36.501874 24494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 03:06:36.505861 24509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:36.507556 24494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:06:36.507798 24494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "dcbe14ad4a8745719f2c2a71b37151ba"
format_stamp: "Formatted at 2026-09-29 03:06:36 on dist-test-slave-530n"
server_key: "6ef950f6bfe8e28e8dedad00e36637d4"
server_key_iv: "22d71e84bdef7344e4384d9c55c9f95d"
server_key_version: "encryptionkey@0"
I20260929 03:06:36.508026 24494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:36.514580 24494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:36.519069 24494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:36.519366 24494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:36.536026 24494 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36289
I20260929 03:06:36.536059 24560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36289 every 8 connection(s)
I20260929 03:06:36.537446 24494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:36.540264 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24494
I20260929 03:06:36.540462 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:36.541054 2870 external_mini_cluster.cc:1428] Setting key 44d37adc95c2c8a4a7c7872ac94c1dfe
I20260929 03:06:36.541710 24561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:36.548552 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:36.542813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56571 (local address 127.0.0.1:36289)
0929 03:06:36.543575 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:36.543587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:36.543628 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:06:36.545140 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:36.545163 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:36.545512 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:36.545835 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:36.546931 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:36.546936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:36.546949 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:36.546972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:36.547073 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:36.547094 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:36.547265 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:36.547407 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:06:36.547613 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260929 03:06:36.551964 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:36.555244 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba: Bootstrap starting.
I20260929 03:06:36.559173 24561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:36.559389 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:36.557160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:32927 (local address 127.0.0.1:36289)
0929 03:06:36.557331 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:36.557336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:36.557563 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:06:36.557723 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:36.557726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:36.557772 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:36.557845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:36.558726 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:36.558731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:36.558734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:36.558747 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:36.558784 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:36.558797 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:36.558876 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:36.559036 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:06:36.559086 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:06:36.560410 24561 log.cc:824] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:36.563294 24561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba: No bootstrap required, opened a new log
I20260929 03:06:36.571102 24561 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260929 03:06:36.571539 24561 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:36.571640 24561 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbe14ad4a8745719f2c2a71b37151ba, State: Initialized, Role: FOLLOWER
I20260929 03:06:36.572256 24561 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260929 03:06:36.572490 24561 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:36.572623 24561 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:36.572825 24561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:36.575114 24561 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260929 03:06:36.575644 24561 leader_election.cc:302] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbe14ad4a8745719f2c2a71b37151ba; no voters:
I20260929 03:06:36.576226 24561 leader_election.cc:288] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:36.576372 24571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:36.577033 24571 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [term 1 LEADER]: Becoming Leader. State: Replica: dcbe14ad4a8745719f2c2a71b37151ba, State: Running, Role: LEADER
I20260929 03:06:36.577625 24571 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260929 03:06:36.578476 24561 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:36.580530 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbe14ad4a8745719f2c2a71b37151ba. Latest consensus state: current_term: 1 leader_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } } }
I20260929 03:06:36.580540 24572 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbe14ad4a8745719f2c2a71b37151ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } } }
I20260929 03:06:36.581004 24572 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:36.581091 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:36.581800 24576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:36.586724 24576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:36.599188 24576 catalog_manager.cc:1347] Generated new cluster ID: a7d8fd3779024691b9d41e4c5e163bf3
I20260929 03:06:36.599298 24576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:36.625528 24576 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:36.626020 24576 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:36.636946 24576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbe14ad4a8745719f2c2a71b37151ba: Generated new TSK 0
I20260929 03:06:36.637871 24576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:36.650249 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24494
W20260929 03:06:36.651340 24556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.15:56571
W20260929 03:06:36.652040 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:36.652165 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24494
2026-09-29T03:06:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:36 dist-test-slave-530n krb5kdc[24592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:36 dist-test-slave-530n krb5kdc[24592](info): set up 2 sockets
Sep 29 03:06:36 dist-test-slave-530n krb5kdc[24592](info): commencing operation
krb5kdc: starting...
W20260929 03:06:37.665251 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:37 dist-test-slave-530n krb5kdc[24592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:37.909535 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36379
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:38.234956 24605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:38.235260 24605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:38.235352 24605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:38.242548 24605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:38.242661 24605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:38.242712 24605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:38.242770 24605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:38.242811 24605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:38.252596 24605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:38.253846 24605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:38.255070 24605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:38.266500 24610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:38.266535 24611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:38.266618 24613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:38.267544 24605 server_base.cc:1034] running on GCE node
I20260929 03:06:38.268252 24605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:38.269719 24605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:38.271059 24605 hybrid_clock.cc:648] HybridClock initialized: now 1790651198271012 us; error 34 us; skew 500 ppm
I20260929 03:06:38.271452 24605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:38 dist-test-slave-530n krb5kdc[24592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:38.274600 24605 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:38.276064 24605 webserver.cc:460] Webserver started at http://127.2.205.190:36457/ using document root <none> and password file <none>
I20260929 03:06:38.276733 24605 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:38.276897 24605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:38.277205 24605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:38.280388 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "793e21f2a91b4723834375101b185446"
format_stamp: "Formatted at 2026-09-29 03:06:38 on dist-test-slave-530n"
server_key: "5cd1fa9e144f8837f8fabe40a49ea77d"
server_key_iv: "56fc50bddf4f39a907f9870e33f9c859"
server_key_version: "encryptionkey@0"
I20260929 03:06:38.281184 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "793e21f2a91b4723834375101b185446"
format_stamp: "Formatted at 2026-09-29 03:06:38 on dist-test-slave-530n"
server_key: "5cd1fa9e144f8837f8fabe40a49ea77d"
server_key_iv: "56fc50bddf4f39a907f9870e33f9c859"
server_key_version: "encryptionkey@0"
I20260929 03:06:38.286481 24605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:06:38.289863 24620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:38.291182 24605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:06:38.291340 24605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "793e21f2a91b4723834375101b185446"
format_stamp: "Formatted at 2026-09-29 03:06:38 on dist-test-slave-530n"
server_key: "5cd1fa9e144f8837f8fabe40a49ea77d"
server_key_iv: "56fc50bddf4f39a907f9870e33f9c859"
server_key_version: "encryptionkey@0"
I20260929 03:06:38.291512 24605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:38.312511 24605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:38.316613 24605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:38.316895 24605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:38.334820 24605 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36379
I20260929 03:06:38.334823 24671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36379 every 8 connection(s)
I20260929 03:06:38.336335 24605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:38.340142 24672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:38.343459 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24605
I20260929 03:06:38.343652 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:38.344185 2870 external_mini_cluster.cc:1428] Setting key 76fbd0b43e65a21dd2d0946a8eb48d57
I20260929 03:06:38.352680 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446: Bootstrap starting.
I20260929 03:06:38.356563 24672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:38.357944 24672 log.cc:824] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:06:38 dist-test-slave-530n krb5kdc[24592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:38.361312 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446: No bootstrap required, opened a new log
I20260929 03:06:38.368067 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:38.346095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50365 (local address 127.2.205.190:36379)
0929 03:06:38.346845 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:38.346858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:38.346900 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:06:38.348606 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:38.348634 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:38.349141 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:38.349501 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:38.351251 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:38.352612 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:38.354389 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:38.355167 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:38.355309 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:38.360400 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:38.360409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:38.360431 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:38.360476 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:38.363473 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:38.364419 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:38.364426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:38.364432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:38.364610 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:38.365170 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:38.365175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:38.365177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:38.365747 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:38.366042 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:38.366606 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 03:06:38.366963 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":346,"threads_started":1}
I20260929 03:06:38.370319 24672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } }
I20260929 03:06:38.370704 24672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:38.370820 24672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793e21f2a91b4723834375101b185446, State: Initialized, Role: FOLLOWER
I20260929 03:06:38.371419 24672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } }
I20260929 03:06:38.371680 24672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:38.371824 24672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:38.372002 24672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:38.372417 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:38.374282 24672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } }
I20260929 03:06:38.374771 24672 leader_election.cc:302] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793e21f2a91b4723834375101b185446; no voters:
I20260929 03:06:38.375452 24672 leader_election.cc:288] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:38.375566 24680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:38.376628 24680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [term 1 LEADER]: Becoming Leader. State: Replica: 793e21f2a91b4723834375101b185446, State: Running, Role: LEADER
I20260929 03:06:38.377266 24680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } }
I20260929 03:06:38.377825 24672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:38.380108 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793e21f2a91b4723834375101b185446. Latest consensus state: current_term: 1 leader_uuid: "793e21f2a91b4723834375101b185446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } } }
I20260929 03:06:38.380314 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793e21f2a91b4723834375101b185446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793e21f2a91b4723834375101b185446" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36379 } } }
I20260929 03:06:38.380576 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:38.380623 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:38.395094 24688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:38.400843 24688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:38.413075 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:38.381033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37381 (local address 127.2.205.190:36379)
0929 03:06:38.381192 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:38.381200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:38.391708 (+ 10508us) server_negotiation.cc:407] Connection header received
0929 03:06:38.392269 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:38.392277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:38.392377 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:38.392471 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:38.397267 (+ 4796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:38.398702 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:38.400860 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:38.401749 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:38.401902 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:38.405049 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:38.405057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:38.405062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:38.405115 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:38.409230 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:38.410325 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:38.410331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:38.410335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:38.410519 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:38.411202 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:38.411208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:38.411211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:38.411677 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:38.411953 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:38.412436 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 03:06:38.412663 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:06:38.413281 24688 catalog_manager.cc:1347] Generated new cluster ID: 606e2a91410c4fc3b6d68eb1c7e828c1
I20260929 03:06:38.413381 24688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:38.435956 24688 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:38.437754 24688 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:38.445884 24688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793e21f2a91b4723834375101b185446: Generated new TSK 0
I20260929 03:06:38.446811 24688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:38.452093 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24605
I20260929 03:06:38.454177 24667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:38.546336 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24605
W20260929 03:06:38.560894 24590 connection.cc:535] client connection to 127.2.205.190:36379 recv error: Network error: failed to read from TLS socket (remote: 127.2.205.190:36379): Connection reset by peer (error 104)
2026-09-29T03:06:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:38 dist-test-slave-530n krb5kdc[24704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:38 dist-test-slave-530n krb5kdc[24704](info): set up 2 sockets
Sep 29 03:06:38 dist-test-slave-530n krb5kdc[24704](info): commencing operation
krb5kdc: starting...
W20260929 03:06:39.565229 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:39 dist-test-slave-530n krb5kdc[24704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:39.807336 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37179
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:40841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:37179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:40.138816 24717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:40.139142 24717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:40.139261 24717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:40.146781 24717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:40.146924 24717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:40.146970 24717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:40.147064 24717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:40.147121 24717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:40.157227 24717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:40841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:37179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:40.158581 24717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:40.159907 24717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:40.171154 24723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:40.171195 24725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:40.171618 24722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:40.172343 24717 server_base.cc:1034] running on GCE node
I20260929 03:06:40.173177 24717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:40.174573 24717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:40.175769 24717 hybrid_clock.cc:648] HybridClock initialized: now 1790651200175730 us; error 26 us; skew 500 ppm
I20260929 03:06:40.176157 24717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:40 dist-test-slave-530n krb5kdc[24704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651200, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:40.179286 24717 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:40.180886 24717 webserver.cc:460] Webserver started at http://127.2.205.190:41809/ using document root <none> and password file <none>
I20260929 03:06:40.181586 24717 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:40.181712 24717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:40.182029 24717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:40.185227 24717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5"
format_stamp: "Formatted at 2026-09-29 03:06:40 on dist-test-slave-530n"
server_key: "e83bbb9b972cbd2105baf77c4b11d921"
server_key_iv: "0919eacf289e7bf98d6b0a163691b18e"
server_key_version: "encryptionkey@0"
I20260929 03:06:40.186029 24717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5"
format_stamp: "Formatted at 2026-09-29 03:06:40 on dist-test-slave-530n"
server_key: "e83bbb9b972cbd2105baf77c4b11d921"
server_key_iv: "0919eacf289e7bf98d6b0a163691b18e"
server_key_version: "encryptionkey@0"
I20260929 03:06:40.191605 24717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:40.195375 24732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:40.196862 24717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:06:40.197047 24717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5"
format_stamp: "Formatted at 2026-09-29 03:06:40 on dist-test-slave-530n"
server_key: "e83bbb9b972cbd2105baf77c4b11d921"
server_key_iv: "0919eacf289e7bf98d6b0a163691b18e"
server_key_version: "encryptionkey@0"
I20260929 03:06:40.197244 24717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:40.227128 24717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:40.231982 24717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:40.232283 24717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:40.250036 24717 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:37179
I20260929 03:06:40.250058 24783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:37179 every 8 connection(s)
I20260929 03:06:40.251394 24717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:40.251695 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24717
I20260929 03:06:40.251839 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:40.252390 2870 external_mini_cluster.cc:1428] Setting key c21191b1bd06970b2f90dd56613bf30b
I20260929 03:06:40.255982 24784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:40.265239 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:40.254263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50397 (local address 127.2.205.190:37179)
0929 03:06:40.255028 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:40.255040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:40.255110 (+ 70us) server_negotiation.cc:407] Connection header received
0929 03:06:40.256822 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:40.256845 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:40.257245 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:40.257594 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:40.258133 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:40.259483 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:40.260798 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:40.261466 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:40.261627 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:40.262883 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:40.262891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:40.262906 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:40.263478 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:40.263553 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:40.263588 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:40.263742 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:40.264019 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:06:40.264190 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":354,"threads_started":1}
I20260929 03:06:40.266808 24784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5: Bootstrap starting.
I20260929 03:06:40.268879 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:40.270000 24784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:40.270985 24784 log.cc:824] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:40.274137 24784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5: No bootstrap required, opened a new log
I20260929 03:06:40.279608 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:40.273005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32931 (local address 127.2.205.190:37179)
0929 03:06:40.273145 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:40.273151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:40.273566 (+ 415us) server_negotiation.cc:407] Connection header received
0929 03:06:40.273742 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:40.273747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:40.273807 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:40.273882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:40.274312 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:40.275418 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:40.276698 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:40.277414 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:40.277556 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:40.278765 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:40.278772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:40.278777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:40.278799 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:40.278856 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:40.278878 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:40.279027 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:40.279203 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 03:06:40.279263 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:06:40.283188 24784 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } }
I20260929 03:06:40.283601 24784 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:40.283677 24784 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cf01c93fd34a53ae77d63ee7f90eb5, State: Initialized, Role: FOLLOWER
I20260929 03:06:40.284271 24784 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } }
I20260929 03:06:40.284488 24784 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:40.284593 24784 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:40.284727 24784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:40.287050 24784 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } }
I20260929 03:06:40.287551 24784 leader_election.cc:302] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7cf01c93fd34a53ae77d63ee7f90eb5; no voters:
I20260929 03:06:40.288116 24784 leader_election.cc:288] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:40.288249 24794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:40.289294 24794 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [term 1 LEADER]: Becoming Leader. State: Replica: e7cf01c93fd34a53ae77d63ee7f90eb5, State: Running, Role: LEADER
I20260929 03:06:40.290120 24794 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } }
I20260929 03:06:40.290280 24784 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:40.293085 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } } }
I20260929 03:06:40.293075 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7cf01c93fd34a53ae77d63ee7f90eb5. Latest consensus state: current_term: 1 leader_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cf01c93fd34a53ae77d63ee7f90eb5" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 37179 } } }
I20260929 03:06:40.293505 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:40.293517 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:40.301497 24800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:40.307265 24800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:40.317085 24800 catalog_manager.cc:1347] Generated new cluster ID: 46451cd1c4934f34b6b4da6e0a4a3853
I20260929 03:06:40.317217 24800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:40.337212 24800 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:40.338711 24800 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:40.347754 24800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7cf01c93fd34a53ae77d63ee7f90eb5: Generated new TSK 0
I20260929 03:06:40.348654 24800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:40.363936 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24717
W20260929 03:06:40.365093 24779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50397
W20260929 03:06:40.365918 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:40.366050 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24717
W20260929 03:06:40.379623 24702 connection.cc:535] client connection to 127.2.205.190:37179 recv error: Network error: failed to read from TLS socket (remote: 127.2.205.190:37179): Connection reset by peer (error 104)
2026-09-29T03:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1818 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:40 dist-test-slave-530n krb5kdc[24815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:40 dist-test-slave-530n krb5kdc[24815](info): set up 2 sockets
Sep 29 03:06:40 dist-test-slave-530n krb5kdc[24815](info): commencing operation
krb5kdc: starting...
W20260929 03:06:41.381305 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:41 dist-test-slave-530n krb5kdc[24815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:41.627836 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45737
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:41.957522 24828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:41.957803 24828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:41.957893 24828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:41.964987 24828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:41.965097 24828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:41.965149 24828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:41.965209 24828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:41.965250 24828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:41.974932 24828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:45737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:41.976135 24828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:41.977435 24828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:41.989275 24836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:41.989323 24834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:41.989302 24833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:41.990099 24828 server_base.cc:1034] running on GCE node
I20260929 03:06:41.990928 24828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:41.992446 24828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:41.993664 24828 hybrid_clock.cc:648] HybridClock initialized: now 1790651201993613 us; error 45 us; skew 500 ppm
I20260929 03:06:41.994024 24828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:42 dist-test-slave-530n krb5kdc[24815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651202, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:42.002254 24828 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:42.004060 24828 webserver.cc:460] Webserver started at http://127.2.205.190:36539/ using document root <none> and password file <none>
I20260929 03:06:42.004886 24828 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:42.005012 24828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:42.005294 24828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:42.008632 24828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "1a486b24f6a24890834dfdb28281ae92"
format_stamp: "Formatted at 2026-09-29 03:06:42 on dist-test-slave-530n"
server_key: "4289eabc656aff153860834dfd76fa2e"
server_key_iv: "0dff116037e23b34543a3a3fb4cc3512"
server_key_version: "encryptionkey@0"
I20260929 03:06:42.009492 24828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "1a486b24f6a24890834dfdb28281ae92"
format_stamp: "Formatted at 2026-09-29 03:06:42 on dist-test-slave-530n"
server_key: "4289eabc656aff153860834dfd76fa2e"
server_key_iv: "0dff116037e23b34543a3a3fb4cc3512"
server_key_version: "encryptionkey@0"
I20260929 03:06:42.015453 24828 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 03:06:42.019289 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:42.020826 24828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:42.021011 24828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "1a486b24f6a24890834dfdb28281ae92"
format_stamp: "Formatted at 2026-09-29 03:06:42 on dist-test-slave-530n"
server_key: "4289eabc656aff153860834dfd76fa2e"
server_key_iv: "0dff116037e23b34543a3a3fb4cc3512"
server_key_version: "encryptionkey@0"
I20260929 03:06:42.021214 24828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:42.027657 24828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:42.032205 24828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:42.032527 24828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:42.050200 24828 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:45737
I20260929 03:06:42.050228 24894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:45737 every 8 connection(s)
I20260929 03:06:42.051558 24828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:42.052443 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24828
I20260929 03:06:42.052639 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:42.053166 2870 external_mini_cluster.cc:1428] Setting key 68a3c0964f40d53f124aa967d75cd004
I20260929 03:06:42.055810 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:42.062284 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:42.055185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45243 (local address 127.2.205.190:45737)
0929 03:06:42.056084 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:42.056099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:42.056395 (+ 296us) server_negotiation.cc:407] Connection header received
0929 03:06:42.058543 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:42.058578 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:42.059011 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:42.059435 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:42.060372 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:42.060380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:42.060398 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:42.060418 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:42.060500 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:42.060526 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:42.060775 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:42.060922 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:06:42.061108 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":353,"threads_started":1}
I20260929 03:06:42.065595 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:42.069154 24895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92: Bootstrap starting.
I20260929 03:06:42.072738 24895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:42.072849 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:42.070478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50555 (local address 127.2.205.190:45737)
0929 03:06:42.070657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:42.070662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:42.071145 (+ 483us) server_negotiation.cc:407] Connection header received
0929 03:06:42.071293 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:42.071297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:42.071341 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:42.071413 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:42.072281 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:42.072285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:42.072288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:42.072300 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:42.072334 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:42.072346 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:42.072425 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:42.072542 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:06:42.072581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:06:42.073830 24895 log.cc:824] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:42.076213 24895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92: No bootstrap required, opened a new log
I20260929 03:06:42.083253 24895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } }
I20260929 03:06:42.083626 24895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:42.083719 24895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a486b24f6a24890834dfdb28281ae92, State: Initialized, Role: FOLLOWER
I20260929 03:06:42.084267 24895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } }
I20260929 03:06:42.084482 24895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:42.084609 24895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:42.084776 24895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:42.087004 24895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } }
I20260929 03:06:42.087489 24895 leader_election.cc:302] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a486b24f6a24890834dfdb28281ae92; no voters:
I20260929 03:06:42.088030 24895 leader_election.cc:288] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:42.088168 24905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:42.089151 24905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [term 1 LEADER]: Becoming Leader. State: Replica: 1a486b24f6a24890834dfdb28281ae92, State: Running, Role: LEADER
I20260929 03:06:42.089704 24905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } }
I20260929 03:06:42.090215 24895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:42.092056 24906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a486b24f6a24890834dfdb28281ae92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } } }
I20260929 03:06:42.092391 24906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:42.092578 24907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a486b24f6a24890834dfdb28281ae92. Latest consensus state: current_term: 1 leader_uuid: "1a486b24f6a24890834dfdb28281ae92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a486b24f6a24890834dfdb28281ae92" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 45737 } } }
I20260929 03:06:42.092933 24907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:42.094635 24911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:42.099790 24911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:42.109128 24911 catalog_manager.cc:1347] Generated new cluster ID: 6f8901f643254d0282e0d3861e7aee95
I20260929 03:06:42.109282 24911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:42.137732 24911 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:42.138334 24911 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:42.149348 24911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a486b24f6a24890834dfdb28281ae92: Generated new TSK 0
I20260929 03:06:42.150449 24911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:42.160001 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24828
W20260929 03:06:42.161147 24890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45243
W20260929 03:06:42.161818 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:42.161949 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24828
2026-09-29T03:06:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1795 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:42 dist-test-slave-530n krb5kdc[24926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:42 dist-test-slave-530n krb5kdc[24926](info): set up 2 sockets
Sep 29 03:06:42 dist-test-slave-530n krb5kdc[24926](info): commencing operation
krb5kdc: starting...
W20260929 03:06:43.165320 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:43 dist-test-slave-530n krb5kdc[24926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:43.410184 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36629
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:36629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:43.736618 24939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:43.737020 24939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:43.737175 24939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:43.744724 24939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:43.744910 24939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:43.744997 24939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:43.745082 24939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:43.745148 24939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:43.755285 24939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:36629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:36629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:43.756616 24939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:43.757961 24939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:43.769102 24944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:43.769104 24945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:43.769141 24947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:43.769917 24939 server_base.cc:1034] running on GCE node
I20260929 03:06:43.770627 24939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:43.771893 24939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:43.773101 24939 hybrid_clock.cc:648] HybridClock initialized: now 1790651203773044 us; error 47 us; skew 500 ppm
I20260929 03:06:43.773425 24939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:43 dist-test-slave-530n krb5kdc[24926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651203, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:43.776471 24939 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:43.777866 24939 webserver.cc:460] Webserver started at http://127.2.205.190:38359/ using document root <none> and password file <none>
I20260929 03:06:43.778527 24939 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:43.778643 24939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:43.778910 24939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:43.781920 24939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "bbc189fc5411434c8728b814e670bcb4"
format_stamp: "Formatted at 2026-09-29 03:06:43 on dist-test-slave-530n"
server_key: "c9de3f4ecb7fc3b06c727f717b07364d"
server_key_iv: "bd5f3449243bead0234fa8e2b1759bfc"
server_key_version: "encryptionkey@0"
I20260929 03:06:43.782740 24939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "bbc189fc5411434c8728b814e670bcb4"
format_stamp: "Formatted at 2026-09-29 03:06:43 on dist-test-slave-530n"
server_key: "c9de3f4ecb7fc3b06c727f717b07364d"
server_key_iv: "bd5f3449243bead0234fa8e2b1759bfc"
server_key_version: "encryptionkey@0"
I20260929 03:06:43.788091 24939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 03:06:43.791654 24954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:43.793227 24939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:43.793403 24939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "bbc189fc5411434c8728b814e670bcb4"
format_stamp: "Formatted at 2026-09-29 03:06:43 on dist-test-slave-530n"
server_key: "c9de3f4ecb7fc3b06c727f717b07364d"
server_key_iv: "bd5f3449243bead0234fa8e2b1759bfc"
server_key_version: "encryptionkey@0"
I20260929 03:06:43.793598 24939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:43.808435 24939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:43.813037 24939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:43.813313 24939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:43.830760 24939 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:36629
I20260929 03:06:43.830778 25005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:36629 every 8 connection(s)
I20260929 03:06:43.832166 24939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:43.834153 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 24939
I20260929 03:06:43.834335 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:43.834918 2870 external_mini_cluster.cc:1428] Setting key e3f41564e155e99a4658555b512d1c67
I20260929 03:06:43.837136 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:06:43 dist-test-slave-530n krb5kdc[24926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:43.850378 25006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: Bootstrap starting.
I20260929 03:06:43.854131 25006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:43.855319 25006 log.cc:824] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:43.858595 25006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: No bootstrap required, opened a new log
I20260929 03:06:43.858896 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:43.836944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35983 (local address 127.2.205.190:36629)
0929 03:06:43.838019 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:43.838035 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:43.838085 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:06:43.840190 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:43.840220 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:43.840856 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:43.841319 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:43.842617 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:43.844633 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:43.846006 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:43.846998 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:43.847185 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:43.851140 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:43.851150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:43.851166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:43.851213 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:43.854842 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:43.855505 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:43.855514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:43.855521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:43.855711 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:43.856248 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:43.856255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:43.856257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:43.856891 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:43.857139 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:43.857561 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 03:06:43.857889 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":485,"threads_started":1}
I20260929 03:06:43.862363 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:43.867651 25006 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } }
I20260929 03:06:43.868064 25006 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:43.868181 25006 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc189fc5411434c8728b814e670bcb4, State: Initialized, Role: FOLLOWER
I20260929 03:06:43.868681 25006 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } }
I20260929 03:06:43.868927 25006 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:43.869045 25006 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:43.869204 25006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:43.871291 25006 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } }
I20260929 03:06:43.871788 25006 leader_election.cc:302] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbc189fc5411434c8728b814e670bcb4; no voters:
I20260929 03:06:43.872414 25006 leader_election.cc:288] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:43.872550 25016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:43.873344 25016 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [term 1 LEADER]: Becoming Leader. State: Replica: bbc189fc5411434c8728b814e670bcb4, State: Running, Role: LEADER
I20260929 03:06:43.873955 25016 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } }
I20260929 03:06:43.874526 25006 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:43.875869 25017 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbc189fc5411434c8728b814e670bcb4. Latest consensus state: current_term: 1 leader_uuid: "bbc189fc5411434c8728b814e670bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } } }
I20260929 03:06:43.876235 25017 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:43.876592 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbc189fc5411434c8728b814e670bcb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc189fc5411434c8728b814e670bcb4" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 36629 } } }
I20260929 03:06:43.876958 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:43.882591 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:43.867209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43813 (local address 127.2.205.190:36629)
0929 03:06:43.867395 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:43.867401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:43.867780 (+ 379us) server_negotiation.cc:407] Connection header received
0929 03:06:43.867934 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:43.867939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:43.868007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:43.868105 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:43.869623 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:43.870364 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:43.871554 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:43.872188 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:43.872321 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:43.875037 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:43.875043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:43.875051 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:43.875084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:43.878975 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:43.879828 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:43.879835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:43.879840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:43.880101 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:43.880576 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:43.880582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:43.880586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:43.881017 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:43.881320 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:43.882118 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 03:06:43.882325 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:06:43.887722 25031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:06:43.887868 25031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:06:43.888018 25021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:43.889768 25021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:43.898114 25021 catalog_manager.cc:1347] Generated new cluster ID: ec54bea46c7544d1b877809f670a1783
I20260929 03:06:43.898253 25021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:43.909440 25021 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:43.911443 25021 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:43.926578 25021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbc189fc5411434c8728b814e670bcb4: Generated new TSK 0
I20260929 03:06:43.927487 25021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:43.942165 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 24939
I20260929 03:06:43.943770 25001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:44.035403 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 24939
2026-09-29T03:06:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:44 dist-test-slave-530n krb5kdc[25038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:44 dist-test-slave-530n krb5kdc[25038](info): set up 2 sockets
Sep 29 03:06:44 dist-test-slave-530n krb5kdc[25038](info): commencing operation
krb5kdc: starting...
W20260929 03:06:45.057221 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:45 dist-test-slave-530n krb5kdc[25038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:45.307616 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35505
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:35505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:45.635694 25051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:45.636013 25051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:45.636165 25051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:45.643446 25051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:45.643569 25051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:45.643648 25051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:45.643730 25051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:45.643792 25051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:45.653690 25051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:35505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:35505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:45.655010 25051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:45.656293 25051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:45.667757 25056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:45.667709 25059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:45.667670 25057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:45.668493 25051 server_base.cc:1034] running on GCE node
I20260929 03:06:45.669303 25051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:45.670730 25051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:45.671931 25051 hybrid_clock.cc:648] HybridClock initialized: now 1790651205671889 us; error 36 us; skew 500 ppm
I20260929 03:06:45.672286 25051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:45 dist-test-slave-530n krb5kdc[25038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651205, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:45.675555 25051 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:45.677134 25051 webserver.cc:460] Webserver started at http://127.2.205.190:40621/ using document root <none> and password file <none>
I20260929 03:06:45.677851 25051 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:45.677958 25051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:45.678251 25051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:45.681281 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "be31623f58744fd4b1acc15e37a77915"
format_stamp: "Formatted at 2026-09-29 03:06:45 on dist-test-slave-530n"
server_key: "22068cde53a097e5a5a874e4b4bf11b8"
server_key_iv: "0393107785f19abc2db77ff7551859b3"
server_key_version: "encryptionkey@0"
I20260929 03:06:45.682011 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "be31623f58744fd4b1acc15e37a77915"
format_stamp: "Formatted at 2026-09-29 03:06:45 on dist-test-slave-530n"
server_key: "22068cde53a097e5a5a874e4b4bf11b8"
server_key_iv: "0393107785f19abc2db77ff7551859b3"
server_key_version: "encryptionkey@0"
I20260929 03:06:45.687067 25051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 03:06:45.690393 25066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:45.691725 25051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:06:45.691877 25051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "be31623f58744fd4b1acc15e37a77915"
format_stamp: "Formatted at 2026-09-29 03:06:45 on dist-test-slave-530n"
server_key: "22068cde53a097e5a5a874e4b4bf11b8"
server_key_iv: "0393107785f19abc2db77ff7551859b3"
server_key_version: "encryptionkey@0"
I20260929 03:06:45.692049 25051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:45.712038 25051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:45.716369 25051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:45.716627 25051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:45.734691 25051 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:35505
I20260929 03:06:45.734706 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:35505 every 8 connection(s)
I20260929 03:06:45.736250 25051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:45.740494 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:45.743232 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25051
I20260929 03:06:45.743418 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:45.743930 2870 external_mini_cluster.cc:1428] Setting key 082ca6f4798abdcf8f825ece9e953b92
I20260929 03:06:45.753203 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915: Bootstrap starting.
I20260929 03:06:45.756871 25118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:45.757972 25118 log.cc:824] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:45.758067 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:45.745966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46905 (local address 127.2.205.190:35505)
0929 03:06:45.746899 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:45.746912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:45.746958 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:06:45.748652 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:45.748677 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:45.749176 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:45.749551 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:45.750160 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:45.751513 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:45.753337 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:45.754074 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:45.754213 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:45.755525 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:45.755533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:45.755549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:45.756121 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:45.756201 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:45.756227 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:45.756312 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:45.756611 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 03:06:45.756965 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":410,"threads_started":1}
I20260929 03:06:45.760914 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915: No bootstrap required, opened a new log
I20260929 03:06:45.761775 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:45.770995 25118 raft_consensus.cc:348] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } }
I20260929 03:06:45.771564 25118 raft_consensus.cc:374] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:45.771696 25118 raft_consensus.cc:729] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be31623f58744fd4b1acc15e37a77915, State: Initialized, Role: FOLLOWER
I20260929 03:06:45.772441 25118 consensus_queue.cc:260] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } }
I20260929 03:06:45.772684 25118 raft_consensus.cc:388] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:45.772837 25118 raft_consensus.cc:482] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:45.773001 25118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:45.774175 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:45.767202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35817 (local address 127.2.205.190:35505)
0929 03:06:45.767414 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:45.767421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:45.767827 (+ 406us) server_negotiation.cc:407] Connection header received
0929 03:06:45.768013 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:45.768019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:45.768111 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:45.768228 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:45.768727 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:45.769877 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:45.771283 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:45.771939 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:45.772073 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:45.773414 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:45.773420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:45.773425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:45.773450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:45.773501 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:45.773522 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:45.773641 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:45.773732 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:06:45.773840 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:06:45.775755 25118 raft_consensus.cc:504] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } }
I20260929 03:06:45.776237 25118 leader_election.cc:302] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be31623f58744fd4b1acc15e37a77915; no voters:
I20260929 03:06:45.776821 25118 leader_election.cc:288] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:45.776962 25128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:45.777598 25128 raft_consensus.cc:686] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [term 1 LEADER]: Becoming Leader. State: Replica: be31623f58744fd4b1acc15e37a77915, State: Running, Role: LEADER
I20260929 03:06:45.778290 25128 consensus_queue.cc:237] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } }
I20260929 03:06:45.780124 25128 sys_catalog.cc:455] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be31623f58744fd4b1acc15e37a77915. Latest consensus state: current_term: 1 leader_uuid: "be31623f58744fd4b1acc15e37a77915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } } }
I20260929 03:06:45.780423 25128 sys_catalog.cc:458] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:45.781635 25130 sys_catalog.cc:455] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be31623f58744fd4b1acc15e37a77915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be31623f58744fd4b1acc15e37a77915" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 35505 } } }
I20260929 03:06:45.781962 25130 sys_catalog.cc:458] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:45.784885 25118 sys_catalog.cc:564] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:45.785070 25132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:45.790511 25132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:45.800424 25132 catalog_manager.cc:1347] Generated new cluster ID: fa6817fdf6524703a470a5b38c9c1a92
I20260929 03:06:45.800531 25132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:45.814958 25132 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:45.816210 25132 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:45.821591 25132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be31623f58744fd4b1acc15e37a77915: Generated new TSK 0
I20260929 03:06:45.822433 25132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:45.830394 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25051
W20260929 03:06:45.832453 25113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46905
W20260929 03:06:45.833328 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:45.833491 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25051
2026-09-29T03:06:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1799 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:45 dist-test-slave-530n krb5kdc[25149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:45 dist-test-slave-530n krb5kdc[25149](info): set up 2 sockets
Sep 29 03:06:45 dist-test-slave-530n krb5kdc[25149](info): commencing operation
krb5kdc: starting...
W20260929 03:06:46.865458 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:47 dist-test-slave-530n krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:47.113899 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:33437
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:41713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:33437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:47.443756 25162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:47.444121 25162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:47.444271 25162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:47.452037 25162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:47.452200 25162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:47.452284 25162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:47.452363 25162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:47.452423 25162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:47.462800 25162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:41713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:33437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:47.464221 25162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:47.465611 25162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:47.476507 25167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:47.476565 25170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:47.476593 25168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:47.477432 25162 server_base.cc:1034] running on GCE node
I20260929 03:06:47.478200 25162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:47.479563 25162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:47.480800 25162 hybrid_clock.cc:648] HybridClock initialized: now 1790651207480730 us; error 31 us; skew 500 ppm
I20260929 03:06:47.481166 25162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:47 dist-test-slave-530n krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:47.487392 25162 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:47.489151 25162 webserver.cc:460] Webserver started at http://127.2.205.190:36343/ using document root <none> and password file <none>
I20260929 03:06:47.489991 25162 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:47.490135 25162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:47.490463 25162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:47.493938 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "7e69eba1dd464c30bc090eba645ce54b"
format_stamp: "Formatted at 2026-09-29 03:06:47 on dist-test-slave-530n"
server_key: "37e50470bc65225724225e0666ebc204"
server_key_iv: "8b210fbb7ce24374464e1f527c8386b9"
server_key_version: "encryptionkey@0"
I20260929 03:06:47.494845 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "7e69eba1dd464c30bc090eba645ce54b"
format_stamp: "Formatted at 2026-09-29 03:06:47 on dist-test-slave-530n"
server_key: "37e50470bc65225724225e0666ebc204"
server_key_iv: "8b210fbb7ce24374464e1f527c8386b9"
server_key_version: "encryptionkey@0"
I20260929 03:06:47.501078 25162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 03:06:47.505028 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:47.506546 25162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:47.506734 25162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "7e69eba1dd464c30bc090eba645ce54b"
format_stamp: "Formatted at 2026-09-29 03:06:47 on dist-test-slave-530n"
server_key: "37e50470bc65225724225e0666ebc204"
server_key_iv: "8b210fbb7ce24374464e1f527c8386b9"
server_key_version: "encryptionkey@0"
I20260929 03:06:47.506953 25162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:47.512883 25162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:47.516994 25162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:47.517268 25162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:47.534292 25162 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:33437
I20260929 03:06:47.534327 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:33437 every 8 connection(s)
I20260929 03:06:47.535591 25162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:47.538615 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25162
I20260929 03:06:47.538806 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:47.539366 2870 external_mini_cluster.cc:1428] Setting key 1dcf2e5a964f087d0e08742c4cc1e82e
I20260929 03:06:47.540095 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:47.548509 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:47.541461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38001 (local address 127.2.205.190:33437)
0929 03:06:47.542249 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:47.542261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:47.542302 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:06:47.545039 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:47.545064 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:47.545452 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:47.545806 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:47.546867 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:47.546873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:47.546890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:47.546907 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:47.546972 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:47.546995 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:47.547182 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:47.547284 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 03:06:47.547469 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":350,"threads_started":1}
I20260929 03:06:47.552445 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:47.554188 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b: Bootstrap starting.
I20260929 03:06:47.557960 25229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:47.559146 25229 log.cc:824] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:47.559969 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:47.557494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36773 (local address 127.2.205.190:33437)
0929 03:06:47.557679 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:47.557685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:47.558103 (+ 418us) server_negotiation.cc:407] Connection header received
0929 03:06:47.558267 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:47.558272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:47.558344 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:47.558465 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:47.559287 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:47.559293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:47.559297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:06:47.559315 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:47.559361 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:06:47.559381 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:47.559496 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:47.559594 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:06:47.559662 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:06:47.562175 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b: No bootstrap required, opened a new log
I20260929 03:06:47.569422 25229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } }
I20260929 03:06:47.569759 25229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:06:47.569832 25229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e69eba1dd464c30bc090eba645ce54b, State: Initialized, Role: FOLLOWER
I20260929 03:06:47.570331 25229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } }
I20260929 03:06:47.570519 25229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:06:47.570611 25229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:06:47.570725 25229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:47.572800 25229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } }
I20260929 03:06:47.573243 25229 leader_election.cc:302] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e69eba1dd464c30bc090eba645ce54b; no voters:
I20260929 03:06:47.573827 25229 leader_election.cc:288] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:06:47.573926 25239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:47.575032 25239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [term 1 LEADER]: Becoming Leader. State: Replica: 7e69eba1dd464c30bc090eba645ce54b, State: Running, Role: LEADER
I20260929 03:06:47.575517 25239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } }
I20260929 03:06:47.575874 25229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:47.578222 25241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e69eba1dd464c30bc090eba645ce54b. Latest consensus state: current_term: 1 leader_uuid: "7e69eba1dd464c30bc090eba645ce54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } } }
I20260929 03:06:47.578583 25241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:47.578819 25240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e69eba1dd464c30bc090eba645ce54b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e69eba1dd464c30bc090eba645ce54b" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 33437 } } }
I20260929 03:06:47.579119 25240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:47.579808 25248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:47.585758 25248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:47.594870 25248 catalog_manager.cc:1347] Generated new cluster ID: 483313bec4ca4e5495d7a6956b1c4d6f
I20260929 03:06:47.595007 25248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:47.623451 25248 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:47.623977 25248 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:47.640563 25248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e69eba1dd464c30bc090eba645ce54b: Generated new TSK 0
I20260929 03:06:47.641773 25248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:47.657258 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25162
W20260929 03:06:47.658437 25224 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38001
W20260929 03:06:47.659121 2870 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:06:47.659256 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25162
2026-09-29T03:06:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1822 ms)
[----------] 18 tests from AuthTokenIssuingTest (33137 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:47 dist-test-slave-530n krb5kdc[25260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:47 dist-test-slave-530n krb5kdc[25260](info): set up 2 sockets
Sep 29 03:06:47 dist-test-slave-530n krb5kdc[25260](info): commencing operation
krb5kdc: starting...
W20260929 03:06:48.669404 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:48 dist-test-slave-530n krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:48.920261 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39275
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:49.251266 25273 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:49.251618 25273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:49.251713 25273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:49.251825 25273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:49.259274 25273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:49.259380 25273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:49.259449 25273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:49.259518 25273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:49.259603 25273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:49.269763 25273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:49.271062 25273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:49.272356 25273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:49.283752 25278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:49.283834 25281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:49.283789 25279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:49.284343 25273 server_base.cc:1034] running on GCE node
I20260929 03:06:49.285117 25273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:49.286490 25273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:49.287716 25273 hybrid_clock.cc:648] HybridClock initialized: now 1790651209287630 us; error 61 us; skew 500 ppm
I20260929 03:06:49.288043 25273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:49 dist-test-slave-530n krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:49.291198 25273 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:49.292799 25273 webserver.cc:460] Webserver started at http://127.2.205.190:38367/ using document root <none> and password file <none>
I20260929 03:06:49.293515 25273 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:49.293629 25273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:49.293953 25273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:49.297355 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "d68a34a6fae3421a8cff5d0f1d9414ac"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "8a2f1bd6380e5c994d366cc6b9560835"
server_key_iv: "d89dba78a0be1b32e44666ec59a4f8c8"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.298237 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "d68a34a6fae3421a8cff5d0f1d9414ac"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "8a2f1bd6380e5c994d366cc6b9560835"
server_key_iv: "d89dba78a0be1b32e44666ec59a4f8c8"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.303802 25273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:06:49.307515 25288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:49.308987 25273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:06:49.309163 25273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "d68a34a6fae3421a8cff5d0f1d9414ac"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "8a2f1bd6380e5c994d366cc6b9560835"
server_key_iv: "d89dba78a0be1b32e44666ec59a4f8c8"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.309377 25273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:49.338577 25273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:49.343061 25273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:49.343333 25273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:49.360437 25273 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:39275
I20260929 03:06:49.360445 25339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:39275 every 8 connection(s)
I20260929 03:06:49.361771 25273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:49.365217 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25273
I20260929 03:06:49.365422 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:49.365561 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:49.366205 2870 external_mini_cluster.cc:1428] Setting key a00531fc122476b3671c46ec937c221f
I20260929 03:06:49.373940 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:49 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.189@KRBTEST.COM" created.
I20260929 03:06:49.397980 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.377547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.2.205.190:39275)
0929 03:06:49.379154 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:49.379163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:49.379215 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:06:49.380864 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:49.380872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:49.381177 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:49.381379 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:49.382525 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.384198 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.385622 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.386420 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.386586 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.390060 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:49.390072 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:49.390091 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:49.390128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:49.393493 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.394292 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.394300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.394302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.394455 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.394869 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.394873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.394875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.395561 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:49.395782 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:49.396428 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 03:06:49.396743 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1418,"spinlock_wait_cycles":79232,"thread_start_us":1206,"threads_started":1}
I20260929 03:06:49.398002 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.377551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39275 (local address 127.0.0.1:36474)
0929 03:06:49.378479 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.378532 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:49.380442 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:49.381395 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:49.381401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:49.381978 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:49.382319 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.382337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.384347 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.384351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.385483 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.385495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.386654 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.386660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.386782 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.387393 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:49.387415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:49.389801 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:49.393632 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.393642 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.393652 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.394149 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.394547 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.394550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.394552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.394704 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.395705 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:49.395712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:49.395975 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 03:06:49.396270 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 03:06:49.396929 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":127,"thread_start_us":371,"threads_started":1}
I20260929 03:06:49.401221 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:49.402419 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.402014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53512)
0929 03:06:49.402185 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.402293 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 03:06:49.402747 25289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
I20260929 03:06:49.404361 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.403957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53516)
0929 03:06:49.404145 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.404224 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 03:06:49.404551 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42307: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
I20260929 03:06:49.427733 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:42307
--webserver_interface=127.2.205.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:06:49.453881 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } attempt: 1
I20260929 03:06:49.455114 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.454660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53524)
0929 03:06:49.454852 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.454966 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:06:49.456053 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.455736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53526)
0929 03:06:49.455860 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.455927 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:06:49.456248 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42307: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
I20260929 03:06:49.524523 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } attempt: 2
I20260929 03:06:49.525804 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.525292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53542)
0929 03:06:49.525538 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.525655 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 03:06:49.526782 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.526425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53552)
0929 03:06:49.526567 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.526651 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:06:49.526958 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42307: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
I20260929 03:06:49.618207 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } attempt: 3
I20260929 03:06:49.619438 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.618969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53556)
0929 03:06:49.619200 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.619297 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 03:06:49.620478 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.620100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53570)
0929 03:06:49.620251 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.620327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:06:49.620679 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42307: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
W20260929 03:06:49.753098 25347 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:49.753436 25347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:49.753527 25347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:49.753635 25347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:49.760825 25347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:49.760919 25347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:49.761003 25347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:49.761070 25347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:49.761157 25347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:49.763952 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } attempt: 4
I20260929 03:06:49.765064 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.764518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53584)
0929 03:06:49.764851 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.764937 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 03:06:49.765981 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.765642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53592)
0929 03:06:49.765784 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.765864 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:06:49.766144 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:42307: Network error: Client connection negotiation failed: client connection to 127.2.205.189:42307: connect: Connection refused (error 111)
I20260929 03:06:49.770999 25347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.205.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:49.772212 25347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:49.773499 25347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:06:49.785446 25347 server_base.cc:1034] running on GCE node
W20260929 03:06:49.785391 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:49.785391 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:49.785808 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:49.786702 25347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:49.788239 25347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:49.789501 25347 hybrid_clock.cc:648] HybridClock initialized: now 1790651209789397 us; error 62 us; skew 500 ppm
I20260929 03:06:49.789881 25347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:49 dist-test-slave-530n krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:49.793286 25347 init.cc:377] Logged in from keytab as kudu/127.2.205.189@KRBTEST.COM (short username kudu)
I20260929 03:06:49.794881 25347 webserver.cc:460] Webserver started at http://127.2.205.189:41821/ using document root <none> and password file <none>
I20260929 03:06:49.795615 25347 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:49.795747 25347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:49.796083 25347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:49.799438 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data/instance:
uuid: "5014b7a38432498bb40b8dd0c7b4723e"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "0328d8383da83c3c415afc6ea28493c5"
server_key_iv: "208d93e2160480f8239df210ae9344b0"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.800294 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal/instance:
uuid: "5014b7a38432498bb40b8dd0c7b4723e"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "0328d8383da83c3c415afc6ea28493c5"
server_key_iv: "208d93e2160480f8239df210ae9344b0"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.805928 25347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:49.809693 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:49.811215 25347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:06:49.811384 25347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "5014b7a38432498bb40b8dd0c7b4723e"
format_stamp: "Formatted at 2026-09-29 03:06:49 on dist-test-slave-530n"
server_key: "0328d8383da83c3c415afc6ea28493c5"
server_key_iv: "208d93e2160480f8239df210ae9344b0"
server_key_version: "encryptionkey@0"
I20260929 03:06:49.811600 25347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:49.831328 25347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:49.835731 25347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:49.836006 25347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:49.853386 25347 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.189:42307
I20260929 03:06:49.853442 25413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.189:42307 every 8 connection(s)
I20260929 03:06:49.854745 25347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
I20260929 03:06:49.858879 25414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:49.862260 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25347
I20260929 03:06:49.862438 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-1/wal/instance
I20260929 03:06:49.862982 2870 external_mini_cluster.cc:1428] Setting key 2902f212178216166b70d64488aeb9ef
I20260929 03:06:49.866815 25414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:49 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.188@KRBTEST.COM; defaulting to no policy
I20260929 03:06:49.887673 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.869763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.2.205.190:39275)
0929 03:06:49.870010 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:49.870017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:49.871217 (+ 1200us) server_negotiation.cc:407] Connection header received
0929 03:06:49.873118 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:49.873124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:49.873197 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:49.873288 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:49.875105 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.875779 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.877786 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.878183 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.878274 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.881834 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:49.881839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:49.881842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:49.881877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:49.884518 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.885327 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.885332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.885334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.885457 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.885880 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.885884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.885886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.886187 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:49.886332 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:49.887099 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 03:06:49.887350 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:06:49.888850 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.870267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39275 (local address 127.0.0.1:36480)
0929 03:06:49.871050 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.871096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:49.872853 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:49.873523 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:49.873533 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:49.874194 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:49.874913 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.874936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.875928 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.875933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.877659 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.877674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.878387 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.878392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.878667 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.879385 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:49.879408 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:49.881590 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:49.884682 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.884692 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.884709 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.885195 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.885561 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.885566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.885570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.885764 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.886311 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:49.886319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:49.886738 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 03:06:49.886986 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 03:06:49.887559 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":326,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.188@KRBTEST.COM" created.
I20260929 03:06:49.890861 25414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:49 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:49.906587 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.891710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53604)
0929 03:06:49.892060 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.892080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:49.892223 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:49.893371 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:49.893375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:49.893819 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:49.893987 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.893997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.895757 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.895761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.896876 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:49.896889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.897976 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.897981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:49.898156 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.898764 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:49.898782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:49.900954 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:49.904091 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.904096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.904101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.904552 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.904947 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:49.904952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:49.904955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:49.905095 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:49.905672 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:49.905676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:49.905810 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:06:49.905938 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:06:49.906224 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 03:06:49.906621 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.891766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.2.205.189:42307)
0929 03:06:49.892830 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:49.892838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:49.892868 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:06:49.892921 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:49.892929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:49.893196 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:49.893360 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:49.894171 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.895619 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.897005 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:49.897824 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:49.897941 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:49.901149 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:49.901168 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:49.901183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:49.901214 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:49.903966 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.904672 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.904679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.904681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.904835 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:49.905215 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:49.905218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:49.905220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:49.905556 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:49.905717 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:49.906058 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 03:06:49.906317 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":858,"thread_start_us":664,"threads_started":1}
I20260929 03:06:49.909049 25414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:49.910144 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.909665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38894)
0929 03:06:49.909832 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.909967 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 03:06:49.910368 25363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:49.911907 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.911513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38906)
0929 03:06:49.911690 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.911781 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 03:06:49.912129 25414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
I20260929 03:06:49.923781 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:45073
--webserver_interface=127.2.205.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:06:49.961447 25414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 1
I20260929 03:06:49.962699 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.962215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38920)
0929 03:06:49.962415 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.962525 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 03:06:49.963721 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:49.963349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38928)
0929 03:06:49.963507 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:49.963590 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:06:49.963928 25414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.032167 25414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 2
I20260929 03:06:50.033356 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.032907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38944)
0929 03:06:50.033083 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.033198 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:06:50.034375 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.034022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38952)
0929 03:06:50.034169 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.034246 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:06:50.034574 25414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.065382 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } attempt: 5
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:50.078456 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.066071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53614)
0929 03:06:50.066267 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.066287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.066398 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.066870 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.066875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.067281 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.067474 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.067484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.068517 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.068522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.069443 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.069457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.070167 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.070173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.070373 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.071033 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.071050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.073093 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.075977 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.075982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.075987 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.076370 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.076738 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.076743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.076771 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.076922 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.077473 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.077477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.077647 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.077781 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.078090 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 03:06:50.078482 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.066339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.2.205.189:42307)
0929 03:06:50.066564 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.066569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.066585 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:06:50.066639 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.066643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.066703 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.066790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.067627 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.068338 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.069591 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.070019 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.070140 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.073280 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.073285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.073289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.073323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.075842 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.076485 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.076489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.076492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.076630 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.077029 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.077034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.077036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.077356 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.077496 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.077946 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.078194 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 03:06:50.079762 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:50.080843 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.080411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38956)
0929 03:06:50.080571 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.080661 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 03:06:50.081845 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.081460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38958)
0929 03:06:50.081641 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.081718 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 03:06:50.082064 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.125821 25414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 3
I20260929 03:06:50.127097 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.126573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38974)
0929 03:06:50.126810 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.126911 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 03:06:50.128419 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.127898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38982)
0929 03:06:50.128169 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.128275 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260929 03:06:50.128655 25414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.133282 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 1
I20260929 03:06:50.134465 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.134011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38988)
0929 03:06:50.134206 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.134299 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 03:06:50.135464 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.135148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:38996)
0929 03:06:50.135277 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.135351 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:06:50.135679 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.203907 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 2
I20260929 03:06:50.205163 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.204629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39006)
0929 03:06:50.204861 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.204950 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:06:50.206215 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.205897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39020)
0929 03:06:50.206018 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.206087 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 03:06:50.206418 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
W20260929 03:06:50.259397 25421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:50.259729 25421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:50.259799 25421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:50.259878 25421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:50.267155 25421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:50.267251 25421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:50.267306 25421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:50.267355 25421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:50.267413 25421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:50.271873 25414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 4
I20260929 03:06:50.273001 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.272533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39026)
0929 03:06:50.272766 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.272856 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 03:06:50.273892 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.273603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39042)
0929 03:06:50.273726 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.273793 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:06:50.274091 25414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.277750 25421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39275,127.2.205.189:42307,127.2.205.188:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.205.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:50.278988 25421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:50.280272 25421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:50.291800 25427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:50.291771 25429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:50.291870 25426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:50.292488 25421 server_base.cc:1034] running on GCE node
I20260929 03:06:50.293347 25421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:50.294881 25421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:50.296113 25421 hybrid_clock.cc:648] HybridClock initialized: now 1790651210296047 us; error 41 us; skew 500 ppm
I20260929 03:06:50.296448 25421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:50.299870 25421 init.cc:377] Logged in from keytab as kudu/127.2.205.188@KRBTEST.COM (short username kudu)
I20260929 03:06:50.301352 25421 webserver.cc:460] Webserver started at http://127.2.205.188:42537/ using document root <none> and password file <none>
I20260929 03:06:50.302158 25421 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:50.302253 25421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:50.302525 25421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:50.305660 25421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data/instance:
uuid: "704b6187657c4719997cebc2a53e210d"
format_stamp: "Formatted at 2026-09-29 03:06:50 on dist-test-slave-530n"
server_key: "ee6a5d47ecd54a210876784cd479fd9d"
server_key_iv: "98634deb40439fcc0052b894316bf3b1"
server_key_version: "encryptionkey@0"
I20260929 03:06:50.306438 25421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal/instance:
uuid: "704b6187657c4719997cebc2a53e210d"
format_stamp: "Formatted at 2026-09-29 03:06:50 on dist-test-slave-530n"
server_key: "ee6a5d47ecd54a210876784cd479fd9d"
server_key_iv: "98634deb40439fcc0052b894316bf3b1"
server_key_version: "encryptionkey@0"
I20260929 03:06:50.311793 25421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 03:06:50.312606 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 3
I20260929 03:06:50.313684 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.313226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39046)
0929 03:06:50.313434 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.313516 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:06:50.314610 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.314278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39058)
0929 03:06:50.314426 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.314510 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:06:50.314827 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:45073: Network error: Client connection negotiation failed: client connection to 127.2.205.188:45073: connect: Connection refused (error 111)
I20260929 03:06:50.316005 25436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:50.317602 25421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:50.317756 25421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "704b6187657c4719997cebc2a53e210d"
format_stamp: "Formatted at 2026-09-29 03:06:50 on dist-test-slave-530n"
server_key: "ee6a5d47ecd54a210876784cd479fd9d"
server_key_iv: "98634deb40439fcc0052b894316bf3b1"
server_key_version: "encryptionkey@0"
I20260929 03:06:50.317950 25421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:50.343925 25421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:50.348372 25421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:50.348605 25421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:50.365097 25421 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.188:45073
I20260929 03:06:50.365168 25487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.188:45073 every 8 connection(s)
I20260929 03:06:50.366339 25421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
I20260929 03:06:50.368096 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25421
I20260929 03:06:50.368251 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650859567927-2870-0/minicluster-data/master-2/wal/instance
I20260929 03:06:50.368659 2870 external_mini_cluster.cc:1428] Setting key c440776dc6ff600b225c5266fe53d7b7
I20260929 03:06:50.370548 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:50.378434 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:50.390374 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.370399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.2.205.188:45073)
0929 03:06:50.371153 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.371164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.371202 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:06:50.372695 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.372716 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.373120 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.373425 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.375098 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.376360 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.378082 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.378759 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.378898 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.383219 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.383234 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.383247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.383295 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.386310 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.386995 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.387002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.387009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.387183 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.387537 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.387541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.387543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.388127 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.388439 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.388879 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.389314 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260929 03:06:50.396138 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.381345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.2.205.190:39275)
0929 03:06:50.381555 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.381561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.382223 (+ 662us) server_negotiation.cc:407] Connection header received
0929 03:06:50.382513 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.382517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.382582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.382678 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.384094 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.384913 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.386349 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.386778 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.386884 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.389969 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.389974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.389977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.390012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.392999 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.393896 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.393902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.393906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.394140 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.394508 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.394512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.394515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.394862 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.394982 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.395607 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.395779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:06:50.396173 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.381343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39275 (local address 127.0.0.1:36484)
0929 03:06:50.382051 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.382097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.382377 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.382750 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.382756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.383606 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.383958 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.383969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.385064 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.385068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.386215 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.386228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.386949 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.386955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.387125 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.387672 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.387689 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.389785 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.393143 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.393149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.393153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.393781 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.394267 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.394271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.394274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.394409 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.394997 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.395005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.395282 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.395486 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.395821 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":332,"threads_started":1}
I20260929 03:06:50.397548 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:50.409036 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.393680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.2.205.190:39275)
0929 03:06:50.394355 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.394359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.394376 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:06:50.394412 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.394415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.394471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.394569 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.395988 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.397385 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.398798 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.399188 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.399295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.403498 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.403503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.403507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.403540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.406320 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.407086 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.407090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.407092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.407217 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.407583 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.407585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.407587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.407858 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.408101 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.408416 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.408636 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":309,"threads_started":1}
I20260929 03:06:50.410315 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.398239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:42307 (local address 127.0.0.1:53628)
0929 03:06:50.398415 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.398434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.398529 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.398831 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.398836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.399207 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.399343 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.399351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.400271 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.400277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.401216 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.401228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.401929 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.401936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.402156 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.402687 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.402711 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.404927 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.407940 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.407946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.407950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.408333 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.408708 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.408713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.408717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.408921 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.409501 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.409506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.409689 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.409800 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.410011 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:06:50.410512 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.398287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.2.205.189:42307)
0929 03:06:50.398458 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.398462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.398479 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:06:50.398644 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.398648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.398709 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.398791 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.399477 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.400143 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.401348 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.401757 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.401874 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.405100 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.405105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.405109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.405144 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.407812 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.408446 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.408451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.408453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.408600 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.409048 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.409054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.409057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.409387 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.409502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.409946 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.410169 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:50.411487 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651210, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:50.425498 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.412405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.2.205.188:45073)
0929 03:06:50.412552 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.412556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.412571 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:06:50.412607 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.412611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.412659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.412733 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.413566 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.414428 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.415935 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.416507 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.416606 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.419730 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.419737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.419742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.419782 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.422662 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.423507 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.423512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.423515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.423675 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.424068 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.424073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.424077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.424367 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.424500 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.424882 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.425097 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:06:50.425603 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.412176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39068)
0929 03:06:50.412342 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.412360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.412453 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.412804 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.412809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.413230 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.413381 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.413389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.414546 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.414552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.415806 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.415818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.416721 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.416728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.416932 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.417425 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.417442 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.419529 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.422819 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.422825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.422830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.423357 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.423813 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.423818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.423822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.423964 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.424484 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.424488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.424634 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.424803 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.425270 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":133}
I20260929 03:06:50.426909 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.411014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.2.205.189:42307)
0929 03:06:50.411163 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.411168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.411247 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:06:50.411401 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.411406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.411468 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.411572 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.413024 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.413933 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.415477 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.415897 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.416021 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.420636 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.420642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.420645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.420680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.423737 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.424522 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.424529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.424533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.424695 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.425147 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.425152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.425155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.425435 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.426108 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.426344 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.426582 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:06:50.430806 25488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: Bootstrap starting.
I20260929 03:06:50.431811 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:50.434567 25488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:50.435624 25488 log.cc:824] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:50.434414 25315 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36488
I20260929 03:06:50.438727 25488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: No bootstrap required, opened a new log
I20260929 03:06:50.446107 25488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.446476 25488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704b6187657c4719997cebc2a53e210d, State: Initialized, Role: FOLLOWER
I20260929 03:06:50.446929 25488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.447795 25495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.448102 25495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.449133 25488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:06:50.457983 25506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:06:50.458101 25506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:06:50.456085 25421 master_runner.cc:186] Error getting master registration for 127.2.205.190:39275: OK, instance_id { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" instance_seqno: 1790651209355763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:06:50.463536 25421 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:06:50.492053 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 4
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:06:50.508812 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.492863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39082)
0929 03:06:50.493067 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.493090 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.493206 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.493597 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.493601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.494314 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.494527 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.494541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.495951 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.495957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.497496 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.497513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.498531 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.498537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.498735 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.499383 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.499412 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.501712 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.505839 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.505843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.505846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.506387 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.506814 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.506819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.506823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.507003 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.507747 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.507752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.507936 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.508077 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.508379 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:06:50.509377 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.492945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.2.205.188:45073)
0929 03:06:50.493230 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.493237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.493260 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:06:50.493325 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.493357 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.493438 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.493547 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.494698 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.495820 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.497651 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.498381 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.498519 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.501920 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.501927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.501932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.501975 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.505680 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.506534 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.506541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.506544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.506701 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.507144 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.507150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.507152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.507627 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.507787 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.508247 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.508485 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 03:06:50.515030 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: Bootstrap starting.
I20260929 03:06:50.519121 25340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:50.520172 25340 log.cc:824] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:50.523162 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: No bootstrap required, opened a new log
I20260929 03:06:50.528205 25340 raft_consensus.cc:348] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.528542 25340 raft_consensus.cc:729] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68a34a6fae3421a8cff5d0f1d9414ac, State: Initialized, Role: FOLLOWER
I20260929 03:06:50.529036 25340 consensus_queue.cc:260] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.529961 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.530364 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.531201 25340 sys_catalog.cc:564] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:50.531314 25315 raft_consensus.cc:482] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:06:50.531533 25315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:50.535118 25315 raft_consensus.cc:504] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.536968 25315 leader_election.cc:288] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [CANDIDATE]: Term 1 election: Requested vote from peers 5014b7a38432498bb40b8dd0c7b4723e (127.2.205.189:42307), 704b6187657c4719997cebc2a53e210d (127.2.205.188:45073)
I20260929 03:06:50.539462 25463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "704b6187657c4719997cebc2a53e210d"
I20260929 03:06:50.540084 25463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:50.544106 25463 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68a34a6fae3421a8cff5d0f1d9414ac in term 1.
I20260929 03:06:50.544938 25289 leader_election.cc:302] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704b6187657c4719997cebc2a53e210d, d68a34a6fae3421a8cff5d0f1d9414ac; no voters:
I20260929 03:06:50.545524 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:50.560927 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [term 1 LEADER]: Becoming Leader. State: Replica: d68a34a6fae3421a8cff5d0f1d9414ac, State: Running, Role: LEADER
I20260929 03:06:50.561625 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.569442 25519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68a34a6fae3421a8cff5d0f1d9414ac. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.569898 25519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:50.573307 25414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } attempt: 5
I20260929 03:06:50.578876 25524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 03:06:50 dist-test-slave-530n krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651209, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:06:50.591897 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.574000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:45073 (local address 127.0.0.1:39098)
0929 03:06:50.574423 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:50.574443 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:50.574577 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:50.575509 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:50.575515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:50.576084 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:50.576311 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.576324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.577942 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.577948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.579287 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:50.579305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.580425 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.580431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:50.580685 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.581401 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:50.581424 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:50.584517 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:50.588846 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.588852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.588857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.589441 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.589932 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:50.589937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:50.589942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:50.590106 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:50.590826 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:50.590832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:50.591003 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 03:06:50.591152 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 03:06:50.591481 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260929 03:06:50.592835 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.574827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.2.205.188:45073)
0929 03:06:50.575017 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.575023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.575044 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:06:50.575111 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.575117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.575198 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.575293 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.576522 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.577738 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.579474 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.580084 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.580211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.585000 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.585007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.585012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.585058 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.588574 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.589604 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.589611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.589616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.589801 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.590228 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.590234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.590238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.590605 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.590759 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.592239 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.592481 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 03:06:50.596438 25294 tablet.cc:2369] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:50.600013 25414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: Bootstrap starting.
W20260929 03:06:50.600759 25442 tablet.cc:2369] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:50.604416 25414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:50.606513 25414 log.cc:824] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:50.613476 25524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:50.616685 25414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: No bootstrap required, opened a new log
I20260929 03:06:50.621371 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.584606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.2.205.190:39275)
0929 03:06:50.584822 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.584828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.599044 (+ 14216us) server_negotiation.cc:407] Connection header received
0929 03:06:50.599351 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.599357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.599441 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.599512 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.601628 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.602736 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.604584 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.605312 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.605419 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.611942 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.611950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.611954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.612003 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.616053 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.617006 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.617011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.617014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.617206 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.617682 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.617687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.617690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.618031 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.618318 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.620691 (+ 2373us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.620973 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:06:50.635669 25414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.638293 25414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5014b7a38432498bb40b8dd0c7b4723e, State: Initialized, Role: FOLLOWER
I20260929 03:06:50.641757 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.584975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.2.205.189:42307)
0929 03:06:50.585164 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.585171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.592505 (+ 7334us) server_negotiation.cc:407] Connection header received
0929 03:06:50.596083 (+ 3578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.596089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.596190 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.596263 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.605638 (+ 9375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.606694 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.618813 (+ 12119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.619533 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.619638 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.628148 (+ 8510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.628156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.628162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.628214 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.632125 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.637037 (+ 4912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.637045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.637049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.637241 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.637772 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.637778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.637782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.638187 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.638446 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.641082 (+ 2636us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.641372 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:06:50.639021 25414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } }
I20260929 03:06:50.652364 25533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.654281 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:50.599488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.2.205.188:45073)
0929 03:06:50.604708 (+ 5220us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.604715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.626700 (+ 21985us) server_negotiation.cc:407] Connection header received
0929 03:06:50.626982 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.626988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.627099 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.627176 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.633428 (+ 6252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.634499 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.639047 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:50.639768 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:50.639882 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:50.645492 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:50.645500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:50.645505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:50.645552 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:50.650661 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.651905 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.651910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.651914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.652066 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:50.652523 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:50.652529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:50.652532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:50.652932 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:50.653160 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:50.653689 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 03:06:50.653929 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5055}
I20260929 03:06:50.654879 25533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.654753 25414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:50.657445 25463 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [term 1 FOLLOWER]: Refusing update from remote peer d68a34a6fae3421a8cff5d0f1d9414ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:50.654824 25389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5014b7a38432498bb40b8dd0c7b4723e"
I20260929 03:06:50.657234 25388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:50.662242 25519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:50.673117 25388 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [term 1 FOLLOWER]: Refusing update from remote peer d68a34a6fae3421a8cff5d0f1d9414ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:50.674077 25519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:50.676710 25389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68a34a6fae3421a8cff5d0f1d9414ac in term 1.
I20260929 03:06:50.663980 25273 master_runner.cc:186] Error getting master registration for 127.2.205.189:42307: OK, instance_id { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" instance_seqno: 1790651209848525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:06:50.681475 25273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:06:50.696602 25495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68a34a6fae3421a8cff5d0f1d9414ac. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.697045 25495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.704393 25534 mvcc.cc:205] Tried to move back new op lower bound from 7334507358837043200 to 7334507358466875392. Current Snapshot: MvccSnapshot[applied={T|T < 7334507358837043200}]
I20260929 03:06:50.714484 25495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.714807 25495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.716681 25533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68a34a6fae3421a8cff5d0f1d9414ac. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.717041 25533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.720300 25537 mvcc.cc:205] Tried to move back new op lower bound from 7334507358837043200 to 7334507358466875392. Current Snapshot: MvccSnapshot[applied={T|T < 7334507358837043200}]
I20260929 03:06:50.726650 25533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.726904 25533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:50.730957 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.731292 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:50.731634 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68a34a6fae3421a8cff5d0f1d9414ac" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39275 } } peers { permanent_uuid: "5014b7a38432498bb40b8dd0c7b4723e" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 42307 } } peers { permanent_uuid: "704b6187657c4719997cebc2a53e210d" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 45073 } } }
I20260929 03:06:50.731858 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:50.732708 25524 catalog_manager.cc:1347] Generated new cluster ID: 524547efc02144fab322b2613cf8c01d
I20260929 03:06:50.732812 25524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:50.782986 25524 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:50.785177 25524 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:50.817555 25524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d68a34a6fae3421a8cff5d0f1d9414ac: Generated new TSK 0
I20260929 03:06:50.818864 25524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:50.839964 25552 catalog_manager.cc:1259] Loaded cluster ID: 524547efc02144fab322b2613cf8c01d
I20260929 03:06:50.840200 25552 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: loading cluster ID for follower catalog manager: success
I20260929 03:06:50.846226 25552 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: acquiring CA information for follower catalog manager: success
I20260929 03:06:50.850401 25552 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5014b7a38432498bb40b8dd0c7b4723e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:06:50.952402 25418 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:50.937608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.2.205.189:42307)
0929 03:06:50.937839 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.937846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.948048 (+ 10202us) server_negotiation.cc:407] Connection header received
0929 03:06:50.950537 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.950544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.950642 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.950721 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.952146 (+ 1425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60634: BlockingRecv error: recv got EOF from 127.0.0.1:60634 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260929 03:06:50.957283 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:50.948597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.2.205.188:45073)
0929 03:06:50.948797 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.948803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.955341 (+ 6538us) server_negotiation.cc:407] Connection header received
0929 03:06:50.955752 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.955758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.955842 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.955907 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.957019 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37638: BlockingRecv error: recv got EOF from 127.0.0.1:37638 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 03:06:50.959492 25493 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:50.945867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.2.205.190:39275)
0929 03:06:50.946059 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:50.946066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:50.954528 (+ 8462us) server_negotiation.cc:407] Connection header received
0929 03:06:50.954684 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:50.954690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:50.954781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:50.954903 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:50.959248 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45482: BlockingRecv error: recv got EOF from 127.0.0.1:45482 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:06:51.020380 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.000658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.2.205.189:42307)
0929 03:06:51.000896 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.000902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.011840 (+ 10938us) server_negotiation.cc:407] Connection header received
0929 03:06:51.012455 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.012462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.012697 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.012866 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.013686 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.014883 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.017247 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.017926 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.018033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.018817 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.019564 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.019688 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.019928 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.020046 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 03:06:51.067421 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.054469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.2.205.188:45073)
0929 03:06:51.054648 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.054655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.065312 (+ 10657us) server_negotiation.cc:407] Connection header received
0929 03:06:51.065835 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.065841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.065999 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.066087 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.067156 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37646: BlockingRecv error: recv got EOF from 127.0.0.1:37646 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:06:51.123174 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.104663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.2.205.189:42307)
0929 03:06:51.104921 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.104928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.116046 (+ 11118us) server_negotiation.cc:407] Connection header received
0929 03:06:51.116417 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.116424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.116521 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.116610 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.117368 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.118513 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.120644 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.121381 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.121486 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.122225 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.122396 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.122458 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.122686 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.122778 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 03:06:51.134879 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.132388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.2.205.188:45073)
0929 03:06:51.132604 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.132611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.133171 (+ 560us) server_negotiation.cc:407] Connection header received
0929 03:06:51.133460 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.133467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.133562 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.133641 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.134656 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37662: BlockingRecv error: recv got EOF from 127.0.0.1:37662 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:06:51.153952 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.146785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.2.205.189:42307)
0929 03:06:51.146960 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.146966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.147494 (+ 528us) server_negotiation.cc:407] Connection header received
0929 03:06:51.147763 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.147770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.147852 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.147911 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.148551 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.149671 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.151760 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.152452 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.152577 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.153144 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.153296 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.153384 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.153543 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.153631 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 03:06:51.164885 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.162389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.2.205.188:45073)
0929 03:06:51.162566 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.162571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.163079 (+ 508us) server_negotiation.cc:407] Connection header received
0929 03:06:51.163371 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.163377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.163471 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.163542 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.164606 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37666: BlockingRecv error: recv got EOF from 127.0.0.1:37666 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:06:51.180256 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.172949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.2.205.189:42307)
0929 03:06:51.173178 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.173184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.173660 (+ 476us) server_negotiation.cc:407] Connection header received
0929 03:06:51.173936 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.173943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.174033 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.174120 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.174786 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.175942 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.177985 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.178637 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.178738 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.179386 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.179531 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.179597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.179830 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.179931 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 03:06:51.190152 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.187595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.2.205.188:45073)
0929 03:06:51.187872 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.187879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.188399 (+ 520us) server_negotiation.cc:407] Connection header received
0929 03:06:51.188688 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.188695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.188821 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.188896 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.189905 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37668: BlockingRecv error: recv got EOF from 127.0.0.1:37668 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260929 03:06:51.208833 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.202672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.2.205.189:42307)
0929 03:06:51.202879 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.202885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.203230 (+ 345us) server_negotiation.cc:407] Connection header received
0929 03:06:51.203535 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.203540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.203620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.203700 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.204206 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.205101 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.207012 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.207513 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.207609 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.208108 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.208230 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.208296 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.208436 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.208504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 03:06:51.220978 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.218922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.2.205.188:45073)
0929 03:06:51.219090 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.219095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.219577 (+ 482us) server_negotiation.cc:407] Connection header received
0929 03:06:51.219731 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.219737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.219836 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.219952 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.220682 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37676: BlockingRecv error: recv got EOF from 127.0.0.1:37676 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:06:51.245872 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.240281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.2.205.189:42307)
0929 03:06:51.240500 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.240507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.241035 (+ 528us) server_negotiation.cc:407] Connection header received
0929 03:06:51.241243 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.241248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.241318 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.241394 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.241822 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.242636 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.243973 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.244424 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.244541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.245081 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.245242 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.245359 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.245482 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.245582 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 03:06:51.253373 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.251137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.2.205.188:45073)
0929 03:06:51.251401 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.251407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.251747 (+ 340us) server_negotiation.cc:407] Connection header received
0929 03:06:51.252022 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.252026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.252105 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.252192 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.253145 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37688: BlockingRecv error: recv got EOF from 127.0.0.1:37688 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 03:06:51.298763 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.292652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.2.205.189:42307)
0929 03:06:51.292906 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.292912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.293269 (+ 357us) server_negotiation.cc:407] Connection header received
0929 03:06:51.293461 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.293468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.293571 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.293709 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.294170 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.295294 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.296722 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.297467 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.297587 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.297961 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.298096 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.298178 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.298310 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.298392 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:06:51.306828 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.304811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.2.205.188:45073)
0929 03:06:51.304998 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.305004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.305348 (+ 344us) server_negotiation.cc:407] Connection header received
0929 03:06:51.305518 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.305524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.305631 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.305737 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.306558 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37690: BlockingRecv error: recv got EOF from 127.0.0.1:37690 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:06:51.379189 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.373989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.2.205.189:42307)
0929 03:06:51.374204 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.374211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.374720 (+ 509us) server_negotiation.cc:407] Connection header received
0929 03:06:51.374918 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.374922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.374991 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.375069 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.375504 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.376189 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.377549 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.377968 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.378076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.378471 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.378588 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.378670 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.378820 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.378904 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 03:06:51.387006 25491 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:51.385223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.2.205.188:45073)
0929 03:06:51.385372 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.385376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.385737 (+ 361us) server_negotiation.cc:407] Connection header received
0929 03:06:51.385893 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.385899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.385997 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.386124 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:51.386794 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37698: BlockingRecv error: recv got EOF from 127.0.0.1:37698 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:06:51.460145 25506 catalog_manager.cc:1259] Loaded cluster ID: 524547efc02144fab322b2613cf8c01d
I20260929 03:06:51.460243 25506 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: loading cluster ID for follower catalog manager: success
I20260929 03:06:51.462715 25506 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: acquiring CA information for follower catalog manager: success
I20260929 03:06:51.464457 25506 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 704b6187657c4719997cebc2a53e210d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:06:51.527467 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.522316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.2.205.189:42307)
0929 03:06:51.522541 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.522548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.522915 (+ 367us) server_negotiation.cc:407] Connection header received
0929 03:06:51.523135 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.523139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.523211 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.523288 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.523717 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.524413 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.525844 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.526240 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.526349 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.526749 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.526874 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.526947 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.527094 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.527179 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:06:51.538478 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.533037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.2.205.188:45073)
0929 03:06:51.533252 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.533258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.533482 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:06:51.533670 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.533676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.533848 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.533947 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.534321 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.535028 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.536384 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.536825 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.536940 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.537476 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.538006 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.538135 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.538174 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.538243 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:06:51.552089 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.544591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.2.205.189:42307)
0929 03:06:51.544865 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.544871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.545765 (+ 894us) server_negotiation.cc:407] Connection header received
0929 03:06:51.546124 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.546130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.546211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.546274 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.547237 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.548247 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.550055 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.550677 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.550791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.551326 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.551460 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.551557 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.551722 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.551811 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:06:51.553090 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.546520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.2.205.188:45073)
0929 03:06:51.546742 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.546749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.547189 (+ 440us) server_negotiation.cc:407] Connection header received
0929 03:06:51.547441 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.547447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.547529 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.547626 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.548183 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.549227 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.551064 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.551706 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.551825 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.552366 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.552484 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.552530 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.552735 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.552840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:06:51.553594 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:51.544431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.2.205.190:39275)
0929 03:06:51.545269 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:51.545274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:51.545819 (+ 545us) server_negotiation.cc:407] Connection header received
0929 03:06:51.546005 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:51.546010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:51.546267 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:51.546443 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:51.546998 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.547968 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.549931 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:51.550535 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:51.550663 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:51.551280 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:51.551725 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:51.551820 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:51.553212 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0929 03:06:51.553302 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":576,"threads_started":1}
I20260929 03:06:51.558614 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25273
I20260929 03:06:51.559556 25335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:51.670153 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25273
I20260929 03:06:51.688114 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25347
I20260929 03:06:51.689322 25409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:51.796828 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25347
I20260929 03:06:51.813716 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25421
I20260929 03:06:51.814807 25483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:51.923493 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25421
2026-09-29T03:06:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4268 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:06:52 dist-test-slave-530n krb5kdc[25666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:06:52 dist-test-slave-530n krb5kdc[25666](info): set up 2 sockets
Sep 29 03:06:52 dist-test-slave-530n krb5kdc[25666](info): commencing operation
krb5kdc: starting...
W20260929 03:06:52.957389 2870 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/test-user.keytab.
Sep 29 03:06:53 dist-test-slave-530n krb5kdc[25666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.205.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
Entry for principal kudu/127.2.205.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab.
I20260929 03:06:53.216050 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39789
--webserver_interface=127.2.205.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:06:53.555961 25679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:53.556321 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:53.556421 25679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:53.556540 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:53.563735 25679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:53.563822 25679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:53.563905 25679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:53.563966 25679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:53.564055 25679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:53.573496 25679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.190.keytab
--principal=kudu/127.2.205.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.190:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.205.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:53.574774 25679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:53.576043 25679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:53.587464 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:53.587441 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:53.587519 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:53.588121 25679 server_base.cc:1034] running on GCE node
I20260929 03:06:53.588936 25679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:53.590346 25679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:53.591563 25679 hybrid_clock.cc:648] HybridClock initialized: now 1790651213591496 us; error 33 us; skew 500 ppm
I20260929 03:06:53.591893 25679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:53 dist-test-slave-530n krb5kdc[25666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:53.594966 25679 init.cc:377] Logged in from keytab as kudu/127.2.205.190@KRBTEST.COM (short username kudu)
I20260929 03:06:53.596490 25679 webserver.cc:460] Webserver started at http://127.2.205.190:35749/ using document root <none> and password file <none>
I20260929 03:06:53.597254 25679 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:53.597354 25679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:53.597656 25679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:53.600667 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data/instance:
uuid: "4e4f53daa826473b839d036c6a63c7fe"
format_stamp: "Formatted at 2026-09-29 03:06:53 on dist-test-slave-530n"
server_key: "9624644cc666a1607091db1ec0d2d925"
server_key_iv: "36ec1b6c8dd63b21872cdf01793e32d2"
server_key_version: "encryptionkey@0"
I20260929 03:06:53.601527 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance:
uuid: "4e4f53daa826473b839d036c6a63c7fe"
format_stamp: "Formatted at 2026-09-29 03:06:53 on dist-test-slave-530n"
server_key: "9624644cc666a1607091db1ec0d2d925"
server_key_iv: "36ec1b6c8dd63b21872cdf01793e32d2"
server_key_version: "encryptionkey@0"
I20260929 03:06:53.607072 25679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:06:53.611076 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:53.612644 25679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:53.612862 25679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal
uuid: "4e4f53daa826473b839d036c6a63c7fe"
format_stamp: "Formatted at 2026-09-29 03:06:53 on dist-test-slave-530n"
server_key: "9624644cc666a1607091db1ec0d2d925"
server_key_iv: "36ec1b6c8dd63b21872cdf01793e32d2"
server_key_version: "encryptionkey@0"
I20260929 03:06:53.613098 25679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:53.635793 25679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:53.640255 25679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:53.640517 25679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:53.657817 25679 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.190:39789
I20260929 03:06:53.657837 25745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.190:39789 every 8 connection(s)
I20260929 03:06:53.659068 25679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/data/info.pb
I20260929 03:06:53.661338 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25679
I20260929 03:06:53.661490 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-0/wal/instance
I20260929 03:06:53.661907 2870 external_mini_cluster.cc:1428] Setting key bc0e4e66ec4c8b4a5abbf134eaf8f30f
I20260929 03:06:53.662667 25746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:53.668829 25746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:53 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.189@KRBTEST.COM" created.
I20260929 03:06:53.693656 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.672034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39789 (local address 127.0.0.1:54824)
0929 03:06:53.672891 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.672937 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:53.675256 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:53.676036 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:53.676042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:53.676789 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:53.677223 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:53.677246 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:53.678764 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:53.678770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:53.680411 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:53.680429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:53.681548 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:53.681556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:53.681761 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:53.682448 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:53.682480 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:53.685252 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:53.689466 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:53.689478 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:53.689489 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:53.689922 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:53.690435 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:53.690439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:53.690441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:53.690599 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:53.691528 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:53.691534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:53.691759 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 03:06:53.691968 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 03:06:53.692410 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260929 03:06:53.693692 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.672026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.2.205.190:39789)
0929 03:06:53.673790 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:53.673796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:53.673835 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:06:53.675604 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:53.675610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:53.675833 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:53.675976 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:53.677428 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:53.678602 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:53.680589 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:53.681278 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:53.681442 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:53.685511 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:53.685520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:53.685535 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:53.685604 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:53.689296 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:53.690090 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:53.690101 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:53.690105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:53.690328 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:53.690710 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:53.690714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:53.690717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:53.691417 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:53.691616 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:53.692158 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 03:06:53.692590 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":1595,"spinlock_wait_cycles":14080,"thread_start_us":1362,"threads_started":1}
I20260929 03:06:53.696828 25746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:53.697940 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.697429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59050)
0929 03:06:53.697651 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.697790 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 03:06:53.698233 25695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
I20260929 03:06:53.699805 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.699367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59060)
0929 03:06:53.699547 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.699646 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:06:53.700047 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:44511: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
Entry for principal kudu/127.2.205.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab.
I20260929 03:06:53.720489 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:44511
--webserver_interface=127.2.205.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:06:53.749396 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } attempt: 1
I20260929 03:06:53.750701 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.750172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59076)
0929 03:06:53.750390 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.750516 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:06:53.751781 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.751376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59084)
0929 03:06:53.751529 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.751627 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 03:06:53.751979 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:44511: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
I20260929 03:06:53.820241 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } attempt: 2
I20260929 03:06:53.821656 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.821069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59090)
0929 03:06:53.821342 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.821460 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 03:06:53.822933 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.822467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59104)
0929 03:06:53.822646 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.822761 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 03:06:53.823173 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:44511: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
I20260929 03:06:53.914505 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } attempt: 3
I20260929 03:06:53.915951 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.915331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59120)
0929 03:06:53.915594 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.915747 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 03:06:53.917243 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:53.916719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59132)
0929 03:06:53.916917 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:53.917030 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 03:06:53.917519 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:44511: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
W20260929 03:06:54.051836 25753 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:54.052177 25753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:54.052249 25753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:54.052335 25753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:54.059528 25753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:54.059612 25753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:54.059665 25753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:54.059713 25753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:54.059770 25753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:54.060827 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } attempt: 4
I20260929 03:06:54.062050 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.061433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59136)
0929 03:06:54.061757 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.061879 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 03:06:54.063082 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.062679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59152)
0929 03:06:54.062832 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.062927 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:06:54.063278 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.189:44511: Network error: Client connection negotiation failed: client connection to 127.2.205.189:44511: connect: Connection refused (error 111)
I20260929 03:06:54.069794 25753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.189.keytab
--principal=kudu/127.2.205.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.189:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.205.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:54.070998 25753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:54.072206 25753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:06:54.083302 25761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:54.083325 25758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:54.083366 25759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:54.084195 25753 server_base.cc:1034] running on GCE node
I20260929 03:06:54.085012 25753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:54.086536 25753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:54.087742 25753 hybrid_clock.cc:648] HybridClock initialized: now 1790651214087671 us; error 61 us; skew 500 ppm
I20260929 03:06:54.088059 25753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:54.091274 25753 init.cc:377] Logged in from keytab as kudu/127.2.205.189@KRBTEST.COM (short username kudu)
I20260929 03:06:54.092941 25753 webserver.cc:460] Webserver started at http://127.2.205.189:35753/ using document root <none> and password file <none>
I20260929 03:06:54.093678 25753 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:54.093772 25753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:54.094103 25753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:54.097288 25753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data/instance:
uuid: "e081a7631ab346d6a0032a693dbc632c"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "cd0a08e76d13c1ab75aae1a2f35b743c"
server_key_iv: "046fc2ad8c2b32e4a1f06e6f19bde7a6"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.098132 25753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal/instance:
uuid: "e081a7631ab346d6a0032a693dbc632c"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "cd0a08e76d13c1ab75aae1a2f35b743c"
server_key_iv: "046fc2ad8c2b32e4a1f06e6f19bde7a6"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.103461 25753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:06:54.106947 25768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:54.108354 25753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:06:54.108518 25753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal
uuid: "e081a7631ab346d6a0032a693dbc632c"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "cd0a08e76d13c1ab75aae1a2f35b743c"
server_key_iv: "046fc2ad8c2b32e4a1f06e6f19bde7a6"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.108734 25753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:54.120664 25753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:54.124840 25753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:54.125056 25753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:54.142248 25753 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.189:44511
I20260929 03:06:54.142230 25819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.189:44511 every 8 connection(s)
I20260929 03:06:54.143721 25753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/data/info.pb
I20260929 03:06:54.145234 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25753
I20260929 03:06:54.145421 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-1/wal/instance
I20260929 03:06:54.146000 2870 external_mini_cluster.cc:1428] Setting key e72022cd4739eb815f80cb88d9715e16
I20260929 03:06:54.148270 25820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:54.156630 25820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.205.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.205.188@KRBTEST.COM" created.
I20260929 03:06:54.178323 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.159702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.2.205.190:39789)
0929 03:06:54.159908 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.159915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.161387 (+ 1472us) server_negotiation.cc:407] Connection header received
0929 03:06:54.163394 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.163401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.163486 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.163574 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.165242 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.165917 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.167497 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.167954 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.168071 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.171890 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.171897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.171902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.171940 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.174848 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.175751 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.175756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.175759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.175908 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.176375 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.176379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.176381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.176701 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.176911 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.177695 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.177958 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:06:54.179462 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.160311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39789 (local address 127.0.0.1:54840)
0929 03:06:54.161204 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.161249 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.163091 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.163846 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.163856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.164425 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.165060 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.165079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.166072 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.166076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.167341 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.167355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.168202 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.168210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.168449 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.169198 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.169226 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.171633 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.175045 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.175053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.175069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.175595 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.176030 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.176034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.176036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.176239 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.176859 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.176868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.177294 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.177568 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.178175 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":377,"threads_started":1}
I20260929 03:06:54.181337 25820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:54.197573 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.181956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59164)
0929 03:06:54.182149 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.182170 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.182281 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.183431 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.183434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.183817 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.183966 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.183974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.185814 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.185819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.186699 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.186711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.187772 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.187776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.187924 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.188556 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.188575 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.190829 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.194911 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.194916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.194920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.195325 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.195799 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.195803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.195806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.195942 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.196739 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.196743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.196910 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.197036 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.197281 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:06:54.197969 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.182239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.2.205.189:44511)
0929 03:06:54.182861 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.182869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.182902 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:06:54.182981 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.182989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.183254 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.183409 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.184132 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.185674 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.186839 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.187636 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.187748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.191075 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.191093 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.191112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.191154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.194733 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.195479 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.195491 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.195494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.195682 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.196057 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.196062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.196066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.196583 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.196859 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.197261 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.197534 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":299,"threads_started":1}
I20260929 03:06:54.200682 25820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:54.201984 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.201509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44718)
0929 03:06:54.201700 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.201839 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 03:06:54.202304 25772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.203874 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.203490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44728)
0929 03:06:54.203631 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.203745 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:06:54.204092 25820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
Entry for principal kudu/127.2.205.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab.
I20260929 03:06:54.208268 2870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
/tmp/dist-test-taskg3SEC6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:43833
--webserver_interface=127.2.205.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.205.148:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:06:54.253458 25820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 1
I20260929 03:06:54.254613 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.254154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44732)
0929 03:06:54.254353 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.254445 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 03:06:54.255694 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.255314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44746)
0929 03:06:54.255501 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.255584 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 03:06:54.255937 25820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.324182 25820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 2
I20260929 03:06:54.325480 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.324948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44754)
0929 03:06:54.325174 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.325286 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 03:06:54.326478 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.326120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44760)
0929 03:06:54.326255 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.326339 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:06:54.326692 25820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.362537 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } attempt: 5
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:54.375912 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.363520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.2.205.189:44511)
0929 03:06:54.363719 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.363725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.363745 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:06:54.363805 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.363808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.363883 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.363969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.364804 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.365525 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.366670 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.367137 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.367249 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.370720 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.370725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.370729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.370766 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.373354 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.374034 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.374038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.374041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.374163 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.374526 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.374530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.374532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.374858 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.374992 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.375368 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.375584 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:06:54.375921 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.363299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59176)
0929 03:06:54.363489 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.363528 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.363657 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.364030 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.364035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.364425 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.364622 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.364633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.365661 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.365666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.366519 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.366533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.367284 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.367290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.367519 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.368213 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.368231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.370493 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.373506 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.373512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.373517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.373910 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.374272 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.374277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.374281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.374420 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.374974 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.374979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.375139 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.375258 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.375572 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 03:06:54.377126 25746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:06:54.378216 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.377805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44770)
0929 03:06:54.377973 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.378072 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 03:06:54.379097 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.378790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44786)
0929 03:06:54.378920 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.378985 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:06:54.379277 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.417934 25820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 3
I20260929 03:06:54.419283 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.418706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44798)
0929 03:06:54.418968 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.419087 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 03:06:54.420508 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.420003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44806)
0929 03:06:54.420260 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.420360 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260929 03:06:54.420738 25820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.430508 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 1
I20260929 03:06:54.431627 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.431166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44820)
0929 03:06:54.431359 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.431453 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:06:54.432693 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.432378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44826)
0929 03:06:54.432506 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.432585 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:06:54.432932 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.501158 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 2
I20260929 03:06:54.502529 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.501887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44840)
0929 03:06:54.502238 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.502360 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260929 03:06:54.503593 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.503254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44844)
0929 03:06:54.503379 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.503466 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:06:54.503805 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
W20260929 03:06:54.536840 25827 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:06:54.537207 25827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:06:54.537278 25827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:06:54.537358 25827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:06:54.544689 25827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:06:54.544824 25827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:06:54.544891 25827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:06:54.544941 25827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:06:54.545006 25827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:06:54.554843 25827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.205.148:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.205.190:39789,127.2.205.189:44511,127.2.205.188:43833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/krb5kdc/kudu_127.2.205.188.keytab
--principal=kudu/127.2.205.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.205.188:43833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.205.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:06:54.556120 25827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:06:54.557407 25827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:06:54.563961 25820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 4
I20260929 03:06:54.564992 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.564502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44846)
0929 03:06:54.564716 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.564839 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 03:06:54.565917 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.565608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44856)
0929 03:06:54.565746 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.565817 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:06:54.566118 25820 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
W20260929 03:06:54.568439 25835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:54.568545 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:06:54.568542 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:06:54.569365 25827 server_base.cc:1034] running on GCE node
I20260929 03:06:54.570068 25827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:06:54.571339 25827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:06:54.572556 25827 hybrid_clock.cc:648] HybridClock initialized: now 1790651214572492 us; error 31 us; skew 500 ppm
I20260929 03:06:54.572881 25827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:06:54.575839 25827 init.cc:377] Logged in from keytab as kudu/127.2.205.188@KRBTEST.COM (short username kudu)
I20260929 03:06:54.577315 25827 webserver.cc:460] Webserver started at http://127.2.205.188:44561/ using document root <none> and password file <none>
I20260929 03:06:54.577973 25827 fs_manager.cc:359] Metadata directory not provided
I20260929 03:06:54.578059 25827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:06:54.578332 25827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:06:54.581408 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data/instance:
uuid: "71a35fb920204c71a65b054416b2d977"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "b5c141041411c16786026a35d6b37036"
server_key_iv: "40179125b6d0ab1e1a2768e47f93ddca"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.582206 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal/instance:
uuid: "71a35fb920204c71a65b054416b2d977"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "b5c141041411c16786026a35d6b37036"
server_key_iv: "40179125b6d0ab1e1a2768e47f93ddca"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.587663 25827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:06:54.591320 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:06:54.592897 25827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 03:06:54.593055 25827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data,/tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal
uuid: "71a35fb920204c71a65b054416b2d977"
format_stamp: "Formatted at 2026-09-29 03:06:54 on dist-test-slave-530n"
server_key: "b5c141041411c16786026a35d6b37036"
server_key_iv: "40179125b6d0ab1e1a2768e47f93ddca"
server_key_version: "encryptionkey@0"
I20260929 03:06:54.593257 25827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:06:54.609970 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 3
I20260929 03:06:54.611186 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.610682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44866)
0929 03:06:54.610911 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.611021 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:06:54.612272 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.611822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44882)
0929 03:06:54.612026 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.612136 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 03:06:54.612468 25746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.205.188:43833: Network error: Client connection negotiation failed: client connection to 127.2.205.188:43833: connect: Connection refused (error 111)
I20260929 03:06:54.618937 25827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:06:54.623450 25827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:06:54.623720 25827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:06:54.640792 25827 rpc_server.cc:307] RPC server started. Bound to: 127.2.205.188:43833
I20260929 03:06:54.640815 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.205.188:43833 every 8 connection(s)
I20260929 03:06:54.642308 25827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/data/info.pb
I20260929 03:06:54.642997 2870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu as pid 25827
I20260929 03:06:54.643183 2870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskg3SEC6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650859567927-2870-0/minicluster-data/master-2/wal/instance
I20260929 03:06:54.643738 2870 external_mini_cluster.cc:1428] Setting key 9feb6b2e3e3beb4dac28401ffc995a1c
I20260929 03:06:54.647079 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:06:54.654017 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
I20260929 03:06:54.665633 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.645728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.2.205.188:43833)
0929 03:06:54.646516 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.646529 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.646573 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:06:54.648251 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.648275 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.648689 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.649128 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.650551 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.651908 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.653196 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.653852 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.654014 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.658109 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.658120 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.658130 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.658166 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.661251 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.661882 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.661888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.661894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.662046 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.662462 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.662466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.662470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.663116 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.663670 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.663963 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.664356 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":340,"threads_started":1}
I20260929 03:06:54.671887 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.656332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.190:39789 (local address 127.0.0.1:54842)
0929 03:06:54.657131 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.657178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.657326 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.657761 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.657768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.658626 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.658964 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.658975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.660313 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.660319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.661975 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.661989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.662870 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.662877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.663055 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.663617 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.663638 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.665722 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.669464 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.669475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.669481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.669999 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.670354 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.670358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.670362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.670517 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.671082 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.671087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.671224 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.671351 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.671552 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":419,"threads_started":1}
I20260929 03:06:54.671962 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.656497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.2.205.190:39789)
0929 03:06:54.656716 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.656722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.657269 (+ 547us) server_negotiation.cc:407] Connection header received
0929 03:06:54.657484 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.657490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.657572 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.657683 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.659122 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.660132 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.662123 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.662697 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.662801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.665915 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.665921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.665926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.665961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.669343 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.670121 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.670124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.670126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.670249 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.670625 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.670630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.670633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.670948 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.671097 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.671503 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.671705 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:06:54.673271 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.190@KRBTEST.COM
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
I20260929 03:06:54.683748 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.668426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.2.205.190:39789)
0929 03:06:54.668986 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.668991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.669015 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:06:54.669059 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.669065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.669133 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.669233 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.670526 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.671619 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.673325 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.673798 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.673923 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.678108 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.678114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.678123 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.678155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.681111 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.681778 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.681784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.681786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.681907 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.682268 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.682271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.682274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.682573 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.682822 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.683165 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.683386 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":222,"threads_started":1}
I20260929 03:06:54.686381 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.673954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.2.205.189:44511)
0929 03:06:54.674124 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.674129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.674148 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:06:54.674204 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.674209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.674275 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.674366 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.675105 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.675743 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.677271 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.677679 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.677774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.681040 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.681045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.681048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.681088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.683524 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.684284 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.684289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.684292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.684466 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.684851 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.684855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.684857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.685124 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.685247 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.685724 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.685975 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:06:54.687072 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.673788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.189:44511 (local address 127.0.0.1:59182)
0929 03:06:54.674007 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.674023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.674110 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.674401 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.674404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.674799 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.674962 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.674974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.675872 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.675877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.677077 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.677089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.677834 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.677840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.678025 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.678607 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.678630 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.680825 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.683700 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.683705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.683709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.684164 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.684564 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.684567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.684569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.684700 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.685241 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.685246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.685437 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.686558 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.686851 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 03:06:54.688041 25894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.205.189@KRBTEST.COM
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.188@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:06:54.700352 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.689037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.2.205.188:43833)
0929 03:06:54.689181 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.689185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.689199 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:06:54.689233 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.689237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.689287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.689353 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.690166 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.690813 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.692019 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.692428 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.692527 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.695457 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.695462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.695465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.695492 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.697831 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.698455 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.698460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.698463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.698573 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.698888 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.698893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.698896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.699175 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.699300 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.699635 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.700017 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":44}
I20260929 03:06:54.700398 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.685504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.2.205.189:44511)
0929 03:06:54.686165 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.686171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.686197 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:06:54.686244 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.686248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.686314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.686431 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.687718 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.688743 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.690318 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.690727 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.690832 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.695046 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.695052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.695055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.695087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.697825 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.698467 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.698473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.698477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.698610 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.698974 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.698979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.698983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.699257 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.699507 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.699798 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.700036 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":325,"threads_started":1}
I20260929 03:06:54.700403 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.688825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44910)
0929 03:06:54.689027 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.689045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.689139 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.689395 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.689400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.689828 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.690014 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.690025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.690932 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.690938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.691899 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.691911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.692571 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.692577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.692794 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.693281 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.693297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.695301 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.697943 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.697948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.697952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.698357 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.698672 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.698676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.698680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.698798 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.699280 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.699284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.699420 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.699541 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.699855 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:06:54.705619 2870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:06:54.706363 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: Bootstrap starting.
I20260929 03:06:54.709702 25894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:54.708066 25721 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e4f53daa826473b839d036c6a63c7fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54858
I20260929 03:06:54.710858 25894 log.cc:824] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:54.713970 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: No bootstrap required, opened a new log
I20260929 03:06:54.723999 25894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.724443 25894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a35fb920204c71a65b054416b2d977, State: Initialized, Role: FOLLOWER
I20260929 03:06:54.725036 25894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.726012 25902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.726346 25902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:54.727247 25894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:06:54.737057 25913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:06:54.737345 25913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:06:54.734951 25827 master_runner.cc:186] Error getting master registration for 127.2.205.190:39789: OK, instance_id { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" instance_seqno: 1790651213652911 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:06:54.743184 25827 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:06:54.789716 25746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 4
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.190@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
I20260929 03:06:54.805413 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.790544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.2.205.188:43833)
0929 03:06:54.790761 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.790766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.790787 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:06:54.790997 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.791003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.791093 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.791203 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.792154 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.793308 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.795136 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.795825 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.795945 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.800026 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.800032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.800035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.800078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.802722 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.803409 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.803414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.803416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.803541 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.803913 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.803917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.803919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.804272 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.804427 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.804841 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.805059 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:06:54.805408 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.790475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44916)
0929 03:06:54.790704 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.790726 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.790866 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.791250 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.791255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.791757 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.791972 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.791986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.793451 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.793458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.794976 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.794994 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.795982 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.795989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.796196 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.796945 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.796969 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.799805 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.802868 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.802873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.802878 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.803287 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.803653 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.803658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.803662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.803806 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.804409 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.804414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.804563 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.804684 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.805011 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 03:06:54.810530 25746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: Bootstrap starting.
I20260929 03:06:54.813926 25746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:54.815017 25746 log.cc:824] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:54.818130 25746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: No bootstrap required, opened a new log
I20260929 03:06:54.825920 25746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.826392 25746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4f53daa826473b839d036c6a63c7fe, State: Initialized, Role: FOLLOWER
I20260929 03:06:54.826894 25746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.827895 25916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.828339 25916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:54.829349 25746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:54.829495 25721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:06:54.829720 25721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:54.833565 25721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.835860 25721 leader_election.cc:288] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [CANDIDATE]: Term 1 election: Requested vote from peers e081a7631ab346d6a0032a693dbc632c (127.2.205.189:44511), 71a35fb920204c71a65b054416b2d977 (127.2.205.188:43833)
I20260929 03:06:54.836342 25869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e4f53daa826473b839d036c6a63c7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71a35fb920204c71a65b054416b2d977"
I20260929 03:06:54.837348 25869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:54.841344 25869 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4f53daa826473b839d036c6a63c7fe in term 1.
I20260929 03:06:54.842133 25698 leader_election.cc:302] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4f53daa826473b839d036c6a63c7fe, 71a35fb920204c71a65b054416b2d977; no voters:
I20260929 03:06:54.842685 25916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:06:54.846653 25916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [term 1 LEADER]: Becoming Leader. State: Replica: 4e4f53daa826473b839d036c6a63c7fe, State: Running, Role: LEADER
I20260929 03:06:54.847298 25916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.862124 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4f53daa826473b839d036c6a63c7fe. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.862537 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:54.863379 25934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:06:54.865316 25820 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } attempt: 5
W20260929 03:06:54.877451 25848 tablet.cc:2369] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:54.884325 25934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:06:54.892398 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.866187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.2.205.190:39789)
0929 03:06:54.866353 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.866359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.872992 (+ 6633us) server_negotiation.cc:407] Connection header received
0929 03:06:54.873453 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.873460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.873543 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.873610 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.876829 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.878164 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.880173 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.880949 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.881060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.884908 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.884916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.884922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.884970 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.888893 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.889845 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.889852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.889856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.890036 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.890519 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.890524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.890528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.890898 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.891141 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.891767 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.892026 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:06:54 dist-test-slave-530n krb5kdc[25666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790651214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.205.189@KRBTEST.COM for kudu/127.2.205.188@KRBTEST.COM
W20260929 03:06:54.893446 25700 tablet.cc:2369] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:06:54.912127 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.865938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.205.188:43833 (local address 127.0.0.1:44938)
0929 03:06:54.866223 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:06:54.866241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:06:54.866354 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:06:54.871419 (+ 5065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:06:54.871425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:06:54.871946 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:06:54.872178 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.872191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.889035 (+ 16844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.889043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.890566 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:06:54.890587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.891967 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.891974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:06:54.892224 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.893171 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:06:54.893198 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:06:54.896103 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:06:54.908896 (+ 12793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.908903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.908908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.909572 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.910124 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:06:54.910130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:06:54.910133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:06:54.910340 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:06:54.911040 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:06:54.911045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:06:54.911218 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:06:54.911363 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 03:06:54.911710 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 03:06:54.901254 25679 master_runner.cc:186] Error getting master registration for 127.2.205.189:44511: OK, instance_id { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" instance_seqno: 1790651214137311 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:06:54.913571 25679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:06:54.913654 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.866641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.2.205.188:43833)
0929 03:06:54.870265 (+ 3624us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.870271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.870298 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:06:54.870366 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.870372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.870462 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.872372 (+ 1910us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.872648 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.888810 (+ 16162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.891032 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.891762 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.892560 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.896340 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.896348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.896356 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.901533 (+ 5177us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.905468 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.909780 (+ 4312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.909786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.909790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.909972 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.910487 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.910492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.910495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.910893 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.912607 (+ 1714us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.912770 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.913038 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3455,"thread_start_us":3258,"threads_started":1}
I20260929 03:06:54.919592 25820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: Bootstrap starting.
I20260929 03:06:54.920123 25869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [term 1 FOLLOWER]: Refusing update from remote peer 4e4f53daa826473b839d036c6a63c7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:54.921237 25916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:54.926057 25820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: Neither blocks nor log segments found. Creating new log.
I20260929 03:06:54.928293 25820 log.cc:824] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:06:54.941895 25820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: No bootstrap required, opened a new log
I20260929 03:06:54.944937 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.889076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.2.205.189:44511)
0929 03:06:54.890893 (+ 1817us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.890898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.898281 (+ 7383us) server_negotiation.cc:407] Connection header received
0929 03:06:54.898352 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.898357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.898439 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.898539 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.904030 (+ 5491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.905127 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.910033 (+ 4906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.910680 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.910794 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.921498 (+ 10704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.921505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.921509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.921551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.931213 (+ 9662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.939672 (+ 8459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.939679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.939683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.939847 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.940818 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.940824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.940828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.941234 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.941498 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.944286 (+ 2788us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.944542 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1645}
I20260929 03:06:54.947154 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:54.865862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.2.205.188:43833)
0929 03:06:54.866046 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:54.866052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:54.891631 (+ 25579us) server_negotiation.cc:407] Connection header received
0929 03:06:54.892627 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:54.892640 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:54.892736 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:54.892839 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:54.906987 (+ 14148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.908115 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.912870 (+ 4755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:54.913549 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:54.913668 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:54.934730 (+ 21062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:06:54.934738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:06:54.934744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:06:54.934792 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:06:54.938482 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.942085 (+ 3603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.942090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.942093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.942299 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:06:54.943196 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:06:54.943201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:06:54.943204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:06:54.943631 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:06:54.943874 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:54.946514 (+ 2640us) server_negotiation.cc:299] Negotiation successful
0929 03:06:54.946785 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:06:54.960281 25820 raft_consensus.cc:348] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.961066 25820 raft_consensus.cc:729] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e081a7631ab346d6a0032a693dbc632c, State: Initialized, Role: FOLLOWER
I20260929 03:06:54.962081 25820 consensus_queue.cc:260] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } }
I20260929 03:06:54.963090 25902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4f53daa826473b839d036c6a63c7fe. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.963409 25902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:54.966548 25939 mvcc.cc:205] Tried to move back new op lower bound from 7334507376237830144 to 7334507376021749760. Current Snapshot: MvccSnapshot[applied={T|T < 7334507376237830144}]
I20260929 03:06:54.967633 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.967869 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:54.969534 25941 mvcc.cc:205] Tried to move back new op lower bound from 7334507376237830144 to 7334507376021749760. Current Snapshot: MvccSnapshot[applied={T|T < 7334507376237830144}]
I20260929 03:06:54.970933 25902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.971159 25902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:54.971908 25934 catalog_manager.cc:1347] Generated new cluster ID: 80d777957cdb47a1bf4fe3701a0d21e6
I20260929 03:06:54.972018 25934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:06:54.974738 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:54.977267 25795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e4f53daa826473b839d036c6a63c7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e081a7631ab346d6a0032a693dbc632c"
I20260929 03:06:54.977849 25795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:06:54.981086 25795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4f53daa826473b839d036c6a63c7fe in term 1.
I20260929 03:06:54.977243 25820 sys_catalog.cc:564] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:06:54.981892 25794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [term 1 FOLLOWER]: Refusing update from remote peer 4e4f53daa826473b839d036c6a63c7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:06:54.982930 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:54.984297 25916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:06:55.007494 25934 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:06:55.009637 25934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:06:55.047914 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4f53daa826473b839d036c6a63c7fe. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:55.048271 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:55.048790 25951 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:55.049243 25951 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:06:55.048573 25927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e4f53daa826473b839d036c6a63c7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4f53daa826473b839d036c6a63c7fe" member_type: VOTER last_known_addr { host: "127.2.205.190" port: 39789 } } peers { permanent_uuid: "e081a7631ab346d6a0032a693dbc632c" member_type: VOTER last_known_addr { host: "127.2.205.189" port: 44511 } } peers { permanent_uuid: "71a35fb920204c71a65b054416b2d977" member_type: VOTER last_known_addr { host: "127.2.205.188" port: 43833 } } }
I20260929 03:06:55.049470 25927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe [sys.catalog]: This master's current role is: LEADER
I20260929 03:06:55.064479 25934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e4f53daa826473b839d036c6a63c7fe: Generated new TSK 0
I20260929 03:06:55.065876 25934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:06:55.140153 25963 catalog_manager.cc:1259] Loaded cluster ID: 80d777957cdb47a1bf4fe3701a0d21e6
I20260929 03:06:55.140277 25963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: loading cluster ID for follower catalog manager: success
I20260929 03:06:55.144120 25963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: acquiring CA information for follower catalog manager: success
I20260929 03:06:55.146780 25963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e081a7631ab346d6a0032a693dbc632c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:06:55.212458 25899 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.198907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.2.205.190:39789)
0929 03:06:55.199137 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.199144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.208512 (+ 9368us) server_negotiation.cc:407] Connection header received
0929 03:06:55.209941 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.209949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.210059 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.210162 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.212110 (+ 1948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54882: BlockingRecv error: recv got EOF from 127.0.0.1:54882 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 03:06:55.216451 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.205759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.2.205.188:43833)
0929 03:06:55.205965 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.205972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.213994 (+ 8022us) server_negotiation.cc:407] Connection header received
0929 03:06:55.214832 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.214838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.214927 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.215002 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.216114 (+ 1112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44940: BlockingRecv error: recv got EOF from 127.0.0.1:44940 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 03:06:55.220943 25900 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.209180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.2.205.189:44511)
0929 03:06:55.209334 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.209339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.218226 (+ 8887us) server_negotiation.cc:407] Connection header received
0929 03:06:55.219100 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.219108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.219275 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.219357 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.220571 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59206: BlockingRecv error: recv got EOF from 127.0.0.1:59206 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:06:55.268988 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.250536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.2.205.189:44511)
0929 03:06:55.250728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.250733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.261393 (+ 10660us) server_negotiation.cc:407] Connection header received
0929 03:06:55.261900 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.261906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.262104 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.262232 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.262925 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.264096 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.266099 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.266789 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.266877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.267558 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.268198 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.268307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.268516 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.268609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:06:55.299028 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.295384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.2.205.188:43833)
0929 03:06:55.295604 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.295612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.297411 (+ 1799us) server_negotiation.cc:407] Connection header received
0929 03:06:55.297680 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.297687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.297810 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.297907 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.298844 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44948: BlockingRecv error: recv got EOF from 127.0.0.1:44948 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 03:06:55.314177 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.308187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.2.205.189:44511)
0929 03:06:55.308396 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.308403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.308948 (+ 545us) server_negotiation.cc:407] Connection header received
0929 03:06:55.309151 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.309158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.309233 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.309315 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.309761 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.310517 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.312142 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.312718 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.312883 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.313348 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.313487 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.313567 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.313763 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.313849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 03:06:55.322782 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.320793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.2.205.188:43833)
0929 03:06:55.320990 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.320997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.321434 (+ 437us) server_negotiation.cc:407] Connection header received
0929 03:06:55.321649 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.321655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.321745 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.321830 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.322540 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44954: BlockingRecv error: recv got EOF from 127.0.0.1:44954 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:06:55.333367 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.327560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.2.205.189:44511)
0929 03:06:55.327819 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.327826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.328269 (+ 443us) server_negotiation.cc:407] Connection header received
0929 03:06:55.328525 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.328531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.328613 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.328690 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.329232 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.330031 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.331465 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.331947 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.332056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.332492 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.332644 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.332723 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.332944 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.333028 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 03:06:55.341202 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.339070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.2.205.188:43833)
0929 03:06:55.339269 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.339275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.339849 (+ 574us) server_negotiation.cc:407] Connection header received
0929 03:06:55.340040 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.340046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.340125 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.340204 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.340953 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44964: BlockingRecv error: recv got EOF from 127.0.0.1:44964 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:06:55.353287 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.347966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.2.205.189:44511)
0929 03:06:55.348234 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.348241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.348807 (+ 566us) server_negotiation.cc:407] Connection header received
0929 03:06:55.348994 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.349001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.349097 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.349175 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.349594 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.350364 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.351679 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.352096 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.352233 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.352578 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.352711 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.352819 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.352935 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.353015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260929 03:06:55.361114 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.358926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.2.205.188:43833)
0929 03:06:55.359242 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.359248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.359738 (+ 490us) server_negotiation.cc:407] Connection header received
0929 03:06:55.359909 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.359914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.359995 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.360092 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.360881 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44980: BlockingRecv error: recv got EOF from 127.0.0.1:44980 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
I20260929 03:06:55.376860 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.371682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.2.205.189:44511)
0929 03:06:55.371868 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.371875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.372337 (+ 462us) server_negotiation.cc:407] Connection header received
0929 03:06:55.372527 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.372532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.372599 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.372682 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.373075 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.373821 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.375127 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.375612 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.375725 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.376095 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.376198 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.376274 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.376406 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.376493 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 03:06:55.384729 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.382526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.2.205.188:43833)
0929 03:06:55.382733 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.382739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.383448 (+ 709us) server_negotiation.cc:407] Connection header received
0929 03:06:55.383642 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.383647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.383722 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.383803 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.384550 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44990: BlockingRecv error: recv got EOF from 127.0.0.1:44990 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:06:55.409789 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.404373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.2.205.189:44511)
0929 03:06:55.404576 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.404583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.405116 (+ 533us) server_negotiation.cc:407] Connection header received
0929 03:06:55.405316 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.405320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.405391 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.405472 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.405898 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.406645 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.408056 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.408514 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.408625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.409070 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.409189 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.409264 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.409405 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.409492 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:06:55.418501 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.416159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.2.205.188:43833)
0929 03:06:55.416366 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.416372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.417103 (+ 731us) server_negotiation.cc:407] Connection header received
0929 03:06:55.417320 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.417326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.417438 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.417527 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.418273 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45006: BlockingRecv error: recv got EOF from 127.0.0.1:45006 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:06:55.458945 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.453688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.2.205.189:44511)
0929 03:06:55.453901 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.453907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.454391 (+ 484us) server_negotiation.cc:407] Connection header received
0929 03:06:55.454576 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.454581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.454664 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.454742 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.455172 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.455938 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.457313 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.457710 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.457817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.458205 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.458342 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.458413 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.458560 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.458648 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 03:06:55.466573 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.464419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.2.205.188:43833)
0929 03:06:55.464607 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.464614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.465325 (+ 711us) server_negotiation.cc:407] Connection header received
0929 03:06:55.465477 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.465483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.465562 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.465645 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.466404 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45018: BlockingRecv error: recv got EOF from 127.0.0.1:45018 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:06:55.541009 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.534409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.2.205.189:44511)
0929 03:06:55.534686 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.534693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.534960 (+ 267us) server_negotiation.cc:407] Connection header received
0929 03:06:55.535153 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.535159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.535263 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.535383 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.535859 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.537140 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.538901 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.539617 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.539751 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.540088 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.540266 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.540433 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.540561 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.540647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 03:06:55.548838 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.546858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.2.205.188:43833)
0929 03:06:55.547038 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.547045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.547403 (+ 358us) server_negotiation.cc:407] Connection header received
0929 03:06:55.547575 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.547580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.547665 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.547747 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.548545 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45022: BlockingRecv error: recv got EOF from 127.0.0.1:45022 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:06:55.685648 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.680313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.2.205.189:44511)
0929 03:06:55.680587 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.680593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.680877 (+ 284us) server_negotiation.cc:407] Connection header received
0929 03:06:55.681111 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.681116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.681200 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.681304 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.681772 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.682540 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.683933 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.684367 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.684479 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.684939 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.685056 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.685134 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.685272 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.685390 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 03:06:55.693089 25897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:06:55.691331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.2.205.188:43833)
0929 03:06:55.691577 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.691584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.691716 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:06:55.691913 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.691917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.691993 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.692081 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:06:55.692910 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45032: BlockingRecv error: recv got EOF from 127.0.0.1:45032 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:06:55.739341 25913 catalog_manager.cc:1259] Loaded cluster ID: 80d777957cdb47a1bf4fe3701a0d21e6
I20260929 03:06:55.739452 25913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: loading cluster ID for follower catalog manager: success
I20260929 03:06:55.742079 25913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: acquiring CA information for follower catalog manager: success
I20260929 03:06:55.744007 25913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71a35fb920204c71a65b054416b2d977: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:06:55.957792 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.952884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.2.205.189:44511)
0929 03:06:55.953125 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.953132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.953371 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:06:55.953510 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.953514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.953593 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.953679 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.954045 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.954779 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.956151 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.956578 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.956684 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.957101 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.957232 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.957301 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.957410 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.957484 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 03:06:55.969755 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.964336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.2.205.188:43833)
0929 03:06:55.964557 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.964564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.964666 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:06:55.964882 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.964888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.965073 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.965173 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.965583 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.966321 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.967619 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.968036 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.968144 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.968638 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.969190 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.969333 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.969412 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.969502 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:06:55.980846 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.975248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.2.205.188:43833)
0929 03:06:55.975436 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.975442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.976429 (+ 987us) server_negotiation.cc:407] Connection header received
0929 03:06:55.976576 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.976582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.976660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.976737 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.977189 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.977850 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.979225 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.979653 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.979768 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.980156 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.980251 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.980321 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.980445 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.980531 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 03:06:55.985292 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.975660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.2.205.189:44511)
0929 03:06:55.975840 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.975846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.980793 (+ 4947us) server_negotiation.cc:407] Connection header received
0929 03:06:55.981048 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.981052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.981118 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.981172 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.981634 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.982395 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.983783 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.984168 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.984270 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.984621 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.984775 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.984853 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.984967 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.985044 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:06:55.992797 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:06:55.974990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.2.205.190:39789)
0929 03:06:55.987694 (+ 12704us) server_negotiation.cc:206] Beginning negotiation
0929 03:06:55.987701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:06:55.987723 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:06:55.987854 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:06:55.987859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:06:55.988094 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:06:55.988302 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:06:55.988797 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.989688 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.991044 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:06:55.991447 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:06:55.991534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:06:55.991926 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:06:55.992269 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:06:55.992354 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:06:55.992433 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:06:55.992497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12562,"thread_start_us":13420,"threads_started":1}
I20260929 03:06:55.996716 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25679
I20260929 03:06:55.997748 25741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:56.108697 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25679
W20260929 03:06:56.126355 25843 connection.cc:535] server connection from 127.0.0.1:44916 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:06:56.126740 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25753
I20260929 03:06:56.127912 25815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:56.236686 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25753
I20260929 03:06:56.254274 2870 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu pid 25827
I20260929 03:06:56.255362 25889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:06:56.363744 2870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskg3SEC6/build/asan/bin/kudu with pid 25827
2026-09-29T03:06:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4439 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8708 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356757 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST